Migrating from ZooKeeper to ClickHouse Keeper

Why teams move a ClickHouse cluster from ZooKeeper to ClickHouse Keeper

This migration replaces the coordination service behind a replicated ClickHouse cluster. ClickHouse uses ZooKeeper to coordinate ReplicatedMergeTree tables, and ClickHouse Keeper is the replacement that the ClickHouse replication guide describes. The move changes which ensemble holds replication metadata, such as table paths, replica state and the part log. It leaves the ReplicatedMergeTree tables, their data parts and the schema in place on the data nodes, so your queries and ingestion keep working against the same tables once the cutover is complete. Plan the move when the existing ZooKeeper ensemble is a maintenance burden, when you are standardizing on Keeper across environments, or when you want to run coordination on the same hosts the ClickHouse team already operates. A move is not a data migration. Nothing about the event or log data is copied to a new location, and no table is rebuilt from scratch. If a source of data is moving between clusters or engines, that work belongs to the migration content group, not this one. Before you pick a window, decide who owns the cutover and who signs off the metadata checks, and confirm the ClickHouse version you run supports the Keeper path you intend to use. Confirm the current release status in the ClickHouse security policy and decide how you will roll back if the converted metadata does not match what the replicas expect. Agree these answers in writing before the window so that the team is not making them under pressure.

Which migration approach fits your ensemble

This table compares three ways to move from a ZooKeeper ensemble to ClickHouse Keeper across host reuse, rollback effort, window drivers and main risk. Read each column against your own host inventory and tolerance for a long maintenance window. The main takeaway is that the more independent the Keeper hosts are from the current ensemble, the easier rollback becomes, at the cost of more infrastructure to run during the move.

Approach	In-place cutover on same hosts	Cutover onto new dedicated Keeper hosts	Staged cutover with warm fallback ensemble
Host reuse	Existing ZooKeeper hosts are reconfigured and reused for Keeper	New hosts are built, typically meeting the dedicated-host guidance for production	Fallback ensemble stays running on its original hosts until the cutover is verified
Rollback effort	Highest, because returning to ZooKeeper means reinstalling or reconfiguring the same machines	Moderate, because the old ensemble can be restarted if the new one is not ready	Lowest, because the warm ensemble can take over again without rebuilding hosts
Maintenance window length drivers	Snapshot conversion, host reconfiguration and restarts on every node in sequence	Snapshot conversion and server configuration changes, with less host work inside the window	Most work happens before the window, but the window still covers the final switch and verification
Background task pausing	Required on all replicas before conversion, so scope is the full cluster	Required on all replicas before conversion, so scope is the full cluster	Required on all replicas before conversion, so scope is the full cluster
Main risk	A failed conversion can leave no working ensemble on the original hosts	Mismatched configuration or hostnames on new hosts causes a failed start	Two ensembles drifting apart or stale metadata if fallback is not kept in sync
Best fit	Small environments with a tight host budget and a rehearsed rollback plan	Most production environments that can add hosts and want a clean rollback path	Regulated or high-impact clusters where a failed window must not block replication

Running the migration in order, with a check at each step

This sequence moves a ClickHouse cluster from ZooKeeper to ClickHouse Keeper in a planned window, with a named check and a go or no-go owner at each stage so that a failed check stops the work before production data is affected.

Confirm version support and read the changelog
Before any planning, the platform lead confirms that the running ClickHouse version appears in the security policy's list of supported versions, and the migration lead reads the changelog for every release between the current version and the target for backward-incompatible changes. The check is a written record of the supported version and a list of changelog items that affect your schema, settings or Keeper configuration. The go or no-go owner is the platform lead. If the current version is unsupported, the upgrade is planned first, because the migration should not be the first change on an unsupported release.
Build and load a pre-production copy
The platform team builds a pre-production cluster with the same ClickHouse version, topology and Keeper layout as production, then loads representative data and runs the queries your teams depend on. The check is a side-by-side comparison of table row counts, replicated table status and the results of key queries against production. The go or no-go owner is the platform lead, with a data owner signing off on the representative data set. A rehearsal on this copy is what exposes problems with converter inputs, paths and hostnames before the real window.
Take and test-restore a backup
The operations team takes a full backup of the affected databases and tables with BACKUP, stores it outside the cluster, and restores it to a spare cluster with RESTORE. The check is that row counts and sample queries on the restored copy match the source. ClickHouse documentation warns that a backup never test-restored will probably not restore properly when needed, so this restore is the gate, not the backup job's success message. The go or no-go owner is the operations lead.
Schedule the window and agree the rollback point
The change owner publishes the maintenance window, the list of servers and Keeper nodes involved, and the rollback procedure, then gets written approval from the application owners who write to these tables. The check is that every named owner confirms the window and that the ZooKeeper ensemble and its snapshots are still intact and reachable. The go or no-go owner is the change owner, who should not start the window without each application owner's sign-off.
Pause background tasks and stop writes
Within the window, engineers stop ingestion jobs and pause background merges and replication fetches as the rehearsed runbook describes, then confirm no inserts or replication queues remain active. The check is that the queue and active-process views show no pending work on any replica and that ingestion sources report no in-flight batches. The go or no-go owner is the on-call database engineer. Any unexplained queue entry is a no-go until it is understood.
Stop ZooKeeper-backed servers and convert the snapshot
Engineers stop the ClickHouse servers that depend on ZooKeeper, keep the ZooKeeper ensemble unchanged for rollback, and run clickhouse-keeper-converter against a copy of the latest ZooKeeper snapshot to produce a ClickHouse Keeper snapshot. The check is that the converter exits without errors and the output snapshot file is recorded with its checksum. The go or no-go owner is the on-call database engineer. Work on a copy so the original ZooKeeper data stays available if the conversion must be repeated.
Start Keeper nodes and point servers at them
Engineers start the ClickHouse Keeper nodes with the converted snapshot, then update each server's configuration so it connects to Keeper rather than ZooKeeper and restart them one at a time. The check is that every Keeper node reports a healthy state, a leader is elected, and each server connects to the ensemble. The go or no-go owner is the platform lead. If any node fails to join or a leader is not elected, stop and follow the rollback path rather than continuing server by server.
Verify metadata, replicas and queries
Engineers resume background work and ingestion in stages, then compare table metadata, replica status, part counts and sample query results against the values recorded before the window. The check is that replicated tables show consistent replica state, no readonly replicas remain, and the verification queries match the pre-window baseline. The go or no-go owner is the change owner, who closes the window only after application owners confirm their workloads. Keep the ZooKeeper ensemble available until this verification and an agreed observation period are complete.

What to verify in the metadata before resuming writes

Conversion alone does not prove the cluster is ready. The checks below confirm that replicated table metadata, replica registration and Keeper quorum match what the team recorded before the window opened. Confirm exact system table names and expected values against the replication documentation for your ClickHouse version before you run them.

Replicated table paths and replica registration

Compare the Keeper paths of every ReplicatedMergeTree table with the paths recorded in the pre-migration inventory, using the shard and replica macros each table expects. Confirm that every replica of each shard is registered under the right path and that no table points at a path left over from ZooKeeper. A missing or duplicated replica name at this stage causes replication to stall once writes resume, so treat any mismatch as a stop condition.

Replication queue and health

After the cluster starts, check the replication queue and replica health views for each table. Look for entries that do not drain, replicas reporting a stale or read-only state, and errors that repeat on the same log entry. Record the queue depth before writes resume so the team can tell normal catch-up from a fault. The exact system table names and healthy values depend on your version, so confirm them in the replication documentation first.

Row and part counts against the baseline

Compare total row counts and active part counts per table with the baseline captured before the maintenance window. Matching totals on every replica of a shard show that conversion did not drop or duplicate parts. Differences are expected only for data written after the baseline was taken, so keep the baseline timestamped. Sample a few high-volume partitions as well, since aggregate counts can hide errors that offset each other.

Keeper quorum and leader status

Confirm that the new Keeper ensemble reports a leader and a quorum of followers, and that every ClickHouse server connects to it. Check that the ensemble survives the loss of one node in a test where the team can safely do so. Keeper health should be verified before any table check, because replica state read from a degraded ensemble can look healthy while it is not.

Test insert and read on each shard

Write a small, labeled test row to a replicated table on each shard, then read it back from every replica of that shard. Repeat the check for one Distributed table if the cluster uses them, so routing and the sharding key are exercised too. Delete the test rows afterward and confirm the counts return to baseline. This is the last check before writes resume, and it proves the path that application traffic will use.

Why background tasks must be paused before the snapshot is taken

A snapshot is only meaningful if the metadata it captures is stable while it is read. In a ZooKeeper-based ClickHouse cluster, ReplicatedMergeTree tables record parts, log entries, replica queues and mutation state as znodes. While merges, mutations, replication fetches and inserts are in flight, those znodes change continuously. The clickhouse-keeper-converter tool reads a ZooKeeper snapshot and writes a Keeper snapshot, so if writes land during that read, the output can describe a state that never existed on any single moment of the cluster, with a part listed in one place and missing in another. A quiet state removes that race. Once the replication queues drain and no new parts are being created, the snapshot represents a consistent starting point that Keeper can load and that replicas can validate against their local data.

Stopping activity is an operator responsibility, and the order matters. Inserts should stop first so that no new parts enter the pipeline, then merges and mutations can be paused so that the set of parts stops changing, and replication fetches should be allowed to finish so that queues drain. Confirm the exact SYSTEM commands, their names and their order against the documentation for the ClickHouse version you run, since the pause and resume options have changed across releases. Before you pause anything, record in a change log which tables, databases and background activities were stopped, on which nodes, and at what time, along with the pre-pause system.replicas and system.replication_queue output. That record is what lets you resume the same set of activities after the conversion, and it is also the baseline you need if you must roll back.

How the cutover differs on Linux hosts and Kubernetes

On bare Linux hosts and package installs, the cutover is a sequence of per-server file and service operations. Each ZooKeeper member is replaced by a Keeper node with its own identity, configuration and data directories, so the team edits and stops services host by host. Confirm the converter invocation, output paths and service names against the versions and packaging your team uses.

Service control: stop ZooKeeper and the ClickHouse server processes through your init system or package manager in the order your runbook defines, and confirm that no process still holds the old data directories.
Config per host: each Keeper node needs its own server identifier, hostname and peer list in its Keeper configuration file, and each ClickHouse replica's Keeper section must point to the new endpoints.
Snapshot placement: copy the converted snapshot only into the snapshot directory of the matching node, and leave the original ZooKeeper data directory untouched so rollback stays possible.
Ownership and permissions: the service account that runs Keeper must own the new log and snapshot paths, since a permissions mismatch is a common reason a node fails to start.
Package pinning: keep the ClickHouse server and Keeper binaries on a matched version set during the window, and hold automatic package updates so the binary does not change mid-cutover.

Before starting the next node, check each host's configuration, service state and snapshot placement against the runbook.

Security controls to keep through the cutover

The cutover temporarily changes who talks to your coordination service and where cluster metadata sits on disk. The cards below cover the controls that must still hold when ZooKeeper is retired, so the new Keeper ensemble is not less protected than the one it replaces.

Encrypted Keeper traffic

Keeper nodes exchange Raft traffic with each other and serve ClickHouse servers, so the ensemble and its client connections need TLS before go-live, not after. Confirm that the Keeper configuration enables secure ports on every node and that ClickHouse servers connect over them. Check that the cutover did not leave a plaintext listener open on the old ports. Ask your security team to review the cipher and certificate settings against your own baseline.

Restricted Keeper port access

During the window both ensembles may be reachable, so firewall rules must allow only ClickHouse servers and peer Keeper nodes to reach the Keeper ports. Scope rules by source address or host group, and remove the ZooKeeper client and peer ports once the new ensemble is confirmed. Verify from outside the cluster that neither ensemble answers on ports it should not expose. Record the rule changes so the rollback path reopens only what it needs.

Service identity and certificates

Each new Keeper node needs a certificate whose identity matches its hostname, and the ClickHouse servers must trust the issuing authority. Plan issuance, renewal and revocation for the new hosts before the window opens, since an expired certificate can stop the ensemble from forming quorum. Use hostnames in the configuration so certificate names and cluster entries stay aligned. Confirm who owns the signing keys and where the private keys are stored.

Protected snapshot and log files

Keeper snapshot and log files hold cluster metadata, including table paths, replica registrations and the replication queue, so they are sensitive even though they contain no row data. Restrict read and write access on the data directories to the Keeper service account, and keep converted snapshots out of shared locations. Apply the same rule to copies made for rollback or troubleshooting, and delete staging copies once verification is complete.

ZooKeeper credential retirement

The old ensemble may hold passwords, digest ACL entries and service accounts that ClickHouse used for replication. Once rollback is no longer needed, remove or rotate those credentials and disable the old ensemble's accounts, so a forgotten secret cannot reopen the coordination layer. Record the date of retirement and the owner who signed off. Confirm the supported version list in the ClickHouse security policy before deciding how long any fallback can remain in place.

Audit of the change window

Record who approved the window, who ran each conversion and configuration step, and which hosts were changed in what order. Keep logs from the operating system, the Keeper and ClickHouse servers, and the change ticket together so that a later question about a replica or table can be traced to a specific action. Store this record outside the cluster it describes. Review it with the team after cutover so that gaps in access or approval are fixed before the next maintenance window.

Questions teams ask before a ZooKeeper to Keeper migration

What does a ZooKeeper to ClickHouse Keeper migration cost, and who pays for what?

The cost is mostly people time, split across three parties. Your engineers spend time on inventory, change approvals, rehearsal and the maintenance window itself. A partner, if you use one, is paid for scoping, the runbook, rehearsal support and window staffing under a written scope. You also pay for a test environment that resembles production, along with the Keeper hosts, which are usually dedicated machines with about 4 GB of RAM per node according to the ClickHouse replication guide. The one-time cost is the migration. The recurring cost is operating a Keeper ensemble you now own, plus any retainer you choose for upgrades and support. Ask any quote to separate these lines.

What makes the migration effort larger or smaller?

The main drivers are the number of ReplicatedMergeTree tables and their Keeper paths, the number of nodes in the cluster, how far your ClickHouse version is from the release you will run, and whether you reuse existing hosts or build new ones. Replicated tables multiply the metadata you must verify afterward. A large version gap adds compatibility checks and possibly an upgrade step. Reusing hosts means extra care with disks and data directories. The quality of your most recent backup restore test also matters, because a backup that has never been restored is an unproven fallback. Ask the quote to state which of these it assumed.

Can our team run this migration without an outside partner?

Yes, if your team has staff who have operated ZooKeeper and ClickHouse replication, a tested rollback path, and time to rehearse on a copy of production. The work is documented, but it is sequenced and unforgiving: pausing background tasks, converting snapshots, checking metadata and deciding when to roll back all happen under pressure. Teams that handle this alone usually need a rehearsal cluster, a named owner for each step and a second person to verify each checkpoint. A partner adds experienced hands and a written runbook, which many teams prefer when the window is short or the cluster is business critical.

How long does the maintenance window need to be, and what sets it?

The window length depends on how many tables and nodes you have, how much metadata the converter must process, how long the rolling restart of each node takes in your environment, and how many verification checks you run before reopening writes. The reliable way to size it is a timed rehearsal on a representative copy of your cluster, not a generic figure. Ask the quote to state the window it plans for, the rehearsal runs behind it and the rollback time if verification fails. Your change calendar and any replication lag you accept will also shape the window.

What do we need to provide before the work starts?

You need administrative access to the ClickHouse servers and existing ZooKeeper ensemble, a current inventory of replicated tables and their Keeper paths, and a recent backup with a restore test result. Change approvals for the rehearsal and the production window must be in place early, and a named decision-maker must be available to call a rollback. For rehearsal, you need representative data and queries, ideally a copy of the production schema and a realistic data volume, so that verification means something. Security reviewers should also confirm how Keeper traffic will be encrypted and which hosts can reach it.

What should we ask for in a written quote?

Ask for the scope as a list of deliverables: inventory, rehearsal plan, runbook, converter steps, verification checks, rollback procedure and post-migration handover. Ask which ClickHouse and Keeper versions the quote assumes and whether an upgrade is included. Ask who staffs the window and at what hours, how many rehearsal runs are included, and what happens if a rehearsal fails. Ask which client inputs are assumed, such as access, approvals and representative data, and what changes if they are late. Finally, ask what recurs after handover, such as monitoring, Keeper patching and an optional support retainer. Fixed-scope engagements are typically agreed in writing before work begins, so the quote should be specific enough to be that agreement.

Limits to confirm for your ClickHouse version and environment

Migrating off ZooKeeper changes the coordination layer under every replicated table, so several limits depend on your exact ClickHouse version, your ensemble layout and the features your schemas use. Work through these checks before scheduling the maintenance window, and treat the official documentation as the authority for each limit.

Conversion path for your exact version

Confirm that your current ClickHouse server version and your target Keeper version both support the snapshot conversion path you plan to use. Support windows differ by release line, and the security policy lists which versions still receive security updates. If your server runs a version outside that list, upgrade first as a separate project, then migrate coordination. Ask DeployOpen to document the exact version pair in the migration plan.

Ensemble size and host layout

Compare your current ZooKeeper ensemble with the replication guidance: at least three Keeper nodes, on dedicated hosts in production, each with about 4 GB of RAM. An ensemble that shares hosts with ClickHouse servers, runs an even number of members, or uses under-provisioned memory changes the failure behavior you are planning around. Record the host layout and decide whether the new ensemble matches it before you cut over.

ZooKeeper-specific behavior in your schemas

Inventory every component that relies on ZooKeeper semantics beyond ReplicatedMergeTree paths, including custom znodes, external tools that read or write ZooKeeper paths, and monitoring that polls ZooKeeper directly. Also review the distributed DDL queue and any ON CLUSTER habits, because those rely on coordination state. Anything that reads ZooKeeper directly must be repointed or retired before the cutover, or it will report stale or empty data.

Rollback to the original ensemble

Confirm whether the conversion you choose can be reversed. Keeping the original ZooKeeper ensemble running and unchanged until verification completes is the simplest rollback path, but it only works if no writes diverged onto the new Keeper. Ask what state changes after the switch, how long you can keep the old ensemble available, and which steps reverse cleanly. Write the rollback trigger and the owner of that decision into the runbook.

Restore tested on a spare cluster

Before the window opens, restore a recent backup of the replicated tables onto a spare cluster and check row counts, checksums and sample queries. The backup documentation warns that a backup never test-restored will probably not restore properly when needed. A successful restore is the fallback if coordination metadata is damaged during conversion, so an untested backup does not count as a rollback plan.

Authoritative limits and version notes

Read the production FAQ and the changelog for your target version before you commit to a date, because minor backward-incompatible changes can occur between releases. Check the release page for the current LTS lines and their published dates. Where the official docs and this checklist differ, the official documentation governs, and you should confirm the limit with ClickHouse or your DeployOpen engineer before proceeding.

How DeployOpen scopes and delivers a Keeper migration

DeployOpen approaches a ZooKeeper to ClickHouse Keeper migration as a written, fixed-scope engagement inside your own infrastructure. We start from your current ensemble, replicated tables and maintenance constraints, then build, test and document the cutover so your team can run the result.

How we scope

Inventory the existing ZooKeeper ensemble, its hosts, paths and every ReplicatedMergeTree table that depends on it.
Agree the maintenance window, rollback point and the background tasks to pause, all written into the scope.
Confirm the target Keeper topology, host placement and the ClickHouse versions the cluster will run during cutover.

How we build

Build a staging copy of the cluster with representative data and rehearse the snapshot conversion with clickhouse-keeper-converter.
Verify replicated table metadata, replica health and queues on the rehearsal before any production step is approved.
Run the production cutover from a documented runbook, with checkpoints where your team confirms each stage before the next starts.

How we hand over

Deliver the runbook, the configuration files, the verification queries and the rollback procedure as versioned documents.
Walk your operators through the cutover record, including what was checked, what changed and what remains open.
Document the Keeper health checks and alerts your team should keep running after the migration closes.

How we support

Offer ongoing support, upgrades and continuous testing as a retainer if your team wants DeployOpen to stay involved.
Agree the post-cutover review window and the support channel in the written scope before work starts.
Ask us to confirm any Keeper or ClickHouse version question against the release notes for your exact version before changes.