ClickHouse cluster discovery with Keeper

How ClickHouse cluster discovery works and when your team needs it

If your team adds or replaces ClickHouse replicas by hand-editing remote_servers on every host, each new node becomes another file to keep in step with the rest of the cluster. Cluster discovery lets new nodes register through ClickHouse Keeper, so the cluster definition is maintained in one place instead of on each server. The need is sharpest when membership changes often, such as replacing a failed host, scaling out a replica set or running nodes that come and go with an autoscaler.

The context is the replication model. Replicated tables use ReplicatedMergeTree engines, and ClickHouse Keeper, which replaces ZooKeeper, coordinates replicas and their metadata. Discovery builds on that same coordination layer, so the Keeper ensemble becomes a dependency for both data replication and cluster membership. Before you adopt it, confirm the Keeper sizing and the health of your ensemble, since a membership problem in Keeper affects more than the registration path. The ClickHouse replication guide describes how Keeper and replicated tables fit together.

The configuration keys, the registration behavior and the supported options depend on the ClickHouse version you run. Confirm the exact mechanics against the current ClickHouse documentation for that version before writing configuration, and test the registration and removal flow in a pre-production cluster with your own topology. Your planning questions include which hosts may register, how a node is removed from the definition when it is retired, and what happens to distributed queries during the window when a node is joining.

Static cluster definitions, Keeper-based discovery and observer nodes compared

This table compares three ways to tell ClickHouse nodes about cluster membership: a static remote_servers list, Keeper-based self-registration, and observer nodes. Read each row against your team's change process, because the main takeaway is that the approach you choose decides who edits configuration every time a node joins or leaves.

Aspect	Static remote_servers on every host	Keeper-based discovery	Observer nodes
What changes operationally	Each node carries the full cluster definition in its configuration files, so membership is fixed until files are updated and servers reloaded	Nodes publish themselves to Keeper, so the shared definition lives in one place that all members read	An observer receives membership and metadata but does not hold replicas, so it adds read or routing capacity without changing quorum
Who edits what when a node is added	The operator edits remote_servers on every existing host and on the new host, then reloads each one	The operator configures the new node to register; existing hosts usually need no change, though the exact settings must be confirmed for your version	The operator configures the observer and points it at the cluster; replicas and Keeper stay untouched
Who edits what when a node is removed	Every remaining host must drop the entry, or stale addresses stay in place and cause failed connections	Confirm how stale registrations are cleared for your version, since a node that stops without deregistering may linger	Remove the observer from its own configuration; replicated data and Keeper membership are not affected
Main risk	Configuration drift: one host with an outdated list can route queries or inserts differently from the rest	Keeper becomes a single point of dependency for membership, and a misconfigured node can register under the wrong identity	Teams may mistake an observer for a replica, leaving data unprotected if it is counted as a copy
Best fit	Small, stable clusters where changes are rare and every host is managed by the same tooling	Clusters that change often, or fleets where hosts are created by automation and cannot be edited by hand	Read-heavy or routing-only roles that need cluster awareness without replica duties
What to confirm before adopting	Your configuration management can push one change to all hosts and verify it landed	The registration settings, key names and defaults in the ClickHouse docs for the version you run	Whether the observer role and its settings are documented for your version and how it interacts with distributed queries

Setting up discovery on an existing ReplicatedMergeTree cluster

This sequence adds node discovery to a cluster that already replicates with ReplicatedMergeTree, one node at a time, so each change is checked before the next. It starts with the Keeper ensemble and ends with two registered nodes that replicate test data. Each step closes with a query or check you can record.

Confirm the Keeper ensemble is healthy
Check the Keeper ensemble before changing any ClickHouse configuration. The replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each, so confirm host placement and memory against that guidance. On each member, send the ruok and mntr commands on the client port and confirm each member reports healthy status, with one leader and followers in sync. The step is done when every member answers and the ensemble has a stable leader. Discovery stores membership in Keeper, so an unhealthy ensemble breaks registration for every node.
Confirm version support and read the docs
Run SELECT version() on every node and compare the result with the discovery documentation for that exact release. Confirm the version is listed as supported with security updates in the ClickHouse security policy. Read the discovery section for your version, because configuration keys can change between releases, and write down the exact key names you will use. Stable releases are recommended for most production use, and the changelog should be read before upgrading because minor backward-incompatible changes can occur. If nodes run mixed versions, align them first and rehearse the change on a pre-production cluster with representative data. The step is done when every node reports the same supported version and the key names are confirmed.
Define the cluster path and node identity
Choose a Keeper path for the cluster that no other cluster or table uses, and plan where the discovery settings sit under the cluster definition in remote_servers. Each node's shard and replica identity should come from macros in its own configuration, checked against the layout you planned. Run SELECT * FROM system.macros on every node and confirm that the shard value matches the planned shard and that replica names are unique within each shard. Done when the macro values match the planned layout and no two nodes share a replica identity. Record the path and the macro table so later steps can be checked against them.
Record the baseline and check replicated tables
Before the first change, save dated copies of the current configuration files on every node. Query system.replicas and confirm that is_readonly is 0 and absolute_delay is low for each replicated table. Check system.replication_queue for stuck entries, and note the table list and row counts for one table as a reference point for later checks. Discovery changes how nodes find each other, so fix any unhealthy table before you start. A fault that appears after the change is harder to attribute to the right cause. The step is done when you hold the configuration copies and a clean replication baseline.
Enable discovery on one existing node
Add the discovery settings to the configuration of one existing node and restart it through your normal change process. Leave the other nodes on their current definitions for now. After the node is back, run SELECT * FROM system.clusters and confirm the cluster lists this node with the expected shard and replica. Then query the Keeper path with SELECT * FROM system.zookeeper WHERE path = '<your path>' and confirm an entry exists for the node. The step is done when the node appears in both places and existing replicated tables still report is_readonly as 0. Keep the restart inside an agreed change window and stop if either check fails.
Add a second node and test replication
Configure the second node with the same discovery path and its own macros, then start it. Confirm that system.clusters lists both hosts under the same shard and that the Keeper path contains two registrations. Create a test ReplicatedMergeTree table with the shared path and macros, insert a set of rows on one node, and read them back from the other node. Watch system.replication_queue until it drains. The step is done when both nodes register, the test rows match on each node, and the queue is empty. Keep the recorded results with your change record, then remove the test table.

Where discovery helps on fixed servers, Kubernetes and autoscaled fleets

On bare-metal or virtual machine clusters, hosts keep the same names and addresses for long periods, so the cluster definition changes mainly when the team adds or retires a machine. Discovery removes the step of editing the cluster definition on every existing host, because a new node can register itself through Keeper. Operators still plan disks, memory, network placement, shard assignment and which replica belongs to which shard. Identity is usually stable here, so the main risk is a hostname that changes after a rebuild.

Removed work: copying the cluster definition to each host and restarting servers for every membership change.
Still manual: disk layout, hardware sizing and deciding which shard a new node serves.
Stable identity: hostnames and replica macros usually survive reboots, so registrations stay valid.
What breaks: a reimaged host that reuses a name but carries different data can attach to the wrong replica path.

Check that hostnames resolve identically from every node and that the replica macros on a rebuilt host match its former identity before it rejoins the cluster.

Security controls for nodes that register themselves

Automatic registration turns Keeper into a membership authority, so whoever can write to its paths can change which hosts the cluster trusts. The controls below work together: limit who can write, protect the traffic, verify each joining node, and record every change.

Who can write to Keeper paths

Registration works by writing a node's entry under a cluster path in Keeper. Any principal allowed to write there can add a host that queries and replicas will then trust. Map which service accounts, automation tools and administrators hold write access to those paths, and remove broad or inherited permissions. Ask your team to confirm the path layout in their own configuration before the first registration runs.

Keeper traffic protection and ACLs

Keeper stores cluster state, so its client and inter-node traffic should be encrypted and authenticated, and its znodes should carry access control lists that match the write rules you chose. Review the ACL and digest settings in the ClickHouse security documentation and confirm them against your Keeper security review. Test that an unauthenticated client cannot read or modify the membership paths.

Verifying a joining node

Discovery should not mean that any host able to reach Keeper becomes a member. Define how a new node proves it was provisioned by you, for example through a host identity issued by your certificate authority or an inventory record checked before registration. Document who approves a join and how a node is removed when it is retired or compromised.

TLS between cluster members

Nodes exchange data and coordinate replication over network connections that can carry sensitive rows and credentials. Use TLS with certificates issued from a trusted authority for both client and replica traffic, and confirm the hostnames in each certificate match the names registered in the cluster. Check how certificate renewal will reach every node without manual edits.

Separating write and read-only membership

Not every host that needs to read cluster metadata should be able to register others. Keep a small set of write-capable identities for provisioning automation, and give query nodes, observers and monitoring agents read-only access. This limits the damage from a compromised application host, which should not be able to add a replica to your cluster.

Audit logging of registration events

Record every registration, change and removal with the identity that made it, the time and the host address. Forward these records to your central log store and alert on registrations that fall outside an approved change window. Test the alert with a deliberately unapproved join in a non-production cluster so you know it fires.

When a static cluster definition is still the better choice

A fixed, small cluster with stable hostnames and infrequent membership changes often gains little from automatic registration. If you add a replica once a year and the hostnames never change, a remote_servers block in a file under version control is usually the simpler design. The explicit definition is easy to read, easy to diff and easy to roll back, and it does not depend on any other component being healthy at the moment a node starts. Before you adopt discovery, ask whether the operational problem it solves actually occurs in your environment, or whether it is a problem you expect to have later.

Discovery also moves part of the cluster's shape into ClickHouse Keeper. Once nodes register themselves there, Keeper becomes a shared dependency for membership changes, not only for replication metadata. If the ensemble is degraded or under maintenance, a new node may fail to join, and the failure can look like a configuration problem on the joining host. Confirm how your team would diagnose a registration that never completes, who has write access to the registry path, and whether that path is covered by the same backup and access review as your replicated tables. These questions matter more than the registration feature itself.

The larger cost is reviewability. A static file in Git shows exactly which hosts belong to which shard and replica, who approved the change and when it took effect. When nodes self-register, that record moves into Keeper state and into the startup logs of each host, so an auditor or an on-call engineer has to reconstruct membership from several places. If your change control or compliance program expects a reviewed record of topology, keep the static definition as the source of truth, or document exactly where the registered state is captured and how it is reviewed.

Limits to confirm before you depend on discovery

Cluster discovery changes who owns cluster membership, so the risks sit in how registrations are added, removed and recovered. The cards below list the behaviors to confirm against the documentation for your exact ClickHouse version and in a pre-production test before any production host depends on them.

Version support and feature status

Confirm whether the discovery feature exists in the version you run, whether it is marked stable, experimental or preview, and which settings it needs. Support windows matter here: the security policy lists 26.9, 26.8, 26.7 and 26.3 as supported, and anything else receives no security updates. Check the release notes for the exact build, not only the release line. Plan an upgrade path if the feature only lands in a newer release.

Stale and failed registration cleanup

Find out how the cluster definition behaves when a node crashes, is replaced or is decommissioned without deregistering. Ask who removes the stale entry, whether it happens automatically, and whether queries or distributed DDL keep targeting a dead host. Test removal of a node during active inserts so you know whether data routed to it is lost, queued or rejected.

Behavior during Keeper quorum loss

Registration writes depend on Keeper, so losing quorum can block new nodes from joining and can freeze the membership view that other nodes read. Confirm which operations stop, which keep serving existing replicas, and whether a node restarted during the outage can still register. The replication guide recommends at least three Keeper nodes on dedicated hosts for production, so test the outage on that layout.

Replica recovery and Keeper membership changes

Verify how a newly registered replica interacts with automatic resync and with the safety check that blocks startup after data loss. Confirm that a rebuilt node registers under the same identity or a new one, because a mismatch can leave orphaned replica paths in Keeper. Also check how dynamic reconfiguration of the Keeper ensemble affects the discovery paths while hosts are being replaced.

Schema for newly registered nodes

Determine whether tables created with ON CLUSTER or through the Replicated database engine appear automatically on a node that registers later. Without that, a new replica can join the cluster definition with no tables, so inserts and distributed queries fail or return partial results. Confirm the behavior for both DDL paths in your environment before you rely on either one.

Backups and monitoring for dynamic hosts

Dynamically added hosts may not appear in backup job lists, scheduled scripts or Prometheus target files that were written for a fixed host inventory. Confirm that each new replica is covered by backup schedules and restore tests, and that metrics and alerts are attached to the registration rather than a hand-edited list. The backup documentation notes that replication alone does not protect against human error or corruption, so backups must follow the nodes that discovery creates.

Questions teams ask before adding cluster discovery

How much effort does it take to move off static remote_servers?

The effort depends mostly on how many clusters, hosts and tables reference the static definition, and on how much of your rollout is already automated. Moving to discovery means setting up the Keeper-backed cluster definition, testing that new nodes register and appear in queries, and then removing the per-host remote_servers entries in a controlled order. Teams with one cluster and a configuration management tool usually have a shorter path than teams with several clusters edited by hand. Ask a partner to list each cluster, host group and configuration file it will touch, so the scope reflects your real inventory rather than a generic estimate.

Can our team do this without outside help?

Yes, if your team already runs ClickHouse Keeper and ReplicatedMergeTree tables and has a staging cluster that mirrors production. The configuration changes themselves are modest, but the risk sits in the sequence: a wrong cluster definition or a missing observer setting can route queries or inserts to the wrong nodes. Outside help is most useful when your team lacks recent experience with Keeper operations, or when the change must be verified against audit or change-control requirements. Either way, the ClickHouse documentation on replication is the starting point for the design decisions your team must own.

What does the work need from our side?

You need to provide four things: a healthy Keeper ensemble with the recommended odd number of nodes on dedicated hosts, a naming convention for hosts and clusters that holds up as the fleet grows, network and credential access to every cluster and Keeper node, and a named owner who can approve changes. Hostnames should be stable and resolvable, because registration depends on identities that survive restarts. Ask your partner to confirm in writing which accounts, firewall rules and change windows the engagement will require before work starts.

What drives the duration of the change?

Duration is driven by cluster count, the number of environments that must be promoted in order (usually development, staging and production), how many configuration sources you maintain, and how thoroughly you test failure cases such as a node restart during registration. Automation that already renders configuration from a source of truth shortens the work, while hand-edited files on each host lengthen it. Ask for the timeline to be broken into phases with explicit exit criteria, so each environment is accepted before the next one changes.

What makes the cost go up or down?

Cost is mostly labor for design, configuration, testing and rollout, with the Keeper hosts and ClickHouse servers already in your budget. Costs rise with the number of clusters and environments, with the amount of custom automation to be replaced, and with the depth of testing required by your change process. Costs fall when the cluster layout is simple, when staging is representative, and when your configuration tooling is already in place. Recurring costs are typically limited to any retainer you choose for upgrades and ongoing support, so ask which items are one-time and which recur.

Does cluster discovery work with Kubernetes operators and autoscaled fleets?

It can help on Kubernetes and autoscaled fleets because new pods or instances can register themselves without a rendered configuration change on every existing host. Whether it fits depends on the operator or deployment tooling you use, how it assigns stable identities, and whether it can supply the macros that replicated tables need. Confirm these points with the operator documentation and your own test cluster before committing to a design. A partner should show, in the written scope, which operator version and deployment method it will validate.

What should we ask a partner to put in a written quote?

Ask for a written scope listing each cluster and environment in scope, the Keeper nodes and ClickHouse hosts that will change, the configuration artifacts to be delivered, and the test cases used to verify registration, query routing and node removal. The quote should also state deliverables such as runbooks, the rollback procedure, the acceptance criteria per environment, and what is one-time versus recurring. Confirm who performs each task, what access they need, and how changes outside the agreed scope are handled. DeployOpen agrees scope, deliverables and timelines in writing before work starts, so you can compare proposals on the same basis.

How does discovery interact with replica recovery?

Discovery changes how nodes are listed in the cluster definition, not how data is repaired. When a replica restarts or loses data, ClickHouse still relies on ReplicatedMergeTree metadata in Keeper, and the documentation notes that a recovered replica automatically syncs missing data from a healthy replica. The practical risk is a node that registers with a stale or reused identity, which can confuse recovery. Your runbooks should cover how a replaced host is assigned its identity, how it is removed from the definition, and how you verify replica health afterward. Keep backups in place as well, since replication does not protect against human error or corruption.

How DeployOpen scopes and delivers cluster discovery work

DeployOpen plans, builds, tests and documents automatic node registration for your self-managed ClickHouse cluster inside infrastructure you control. Each engagement starts with a written scope that fixes which nodes, Keeper ensemble and cluster definitions are in play, so your team knows what is delivered before work begins.

How we scope

We confirm which clusters, shards and replicas will use Keeper-based cluster definitions and which stay in static configuration.
We map the existing Keeper ensemble, its hostnames and its security settings, and record any changes needed before discovery is enabled.
We agree in writing how node onboarding, observer nodes and Kubernetes or autoscaling triggers are covered, and what sits outside scope.

How we build

We define the cluster discovery path in Keeper and configure each server to read it, with the settings checked against your ClickHouse version.
We test joining, leaving and restarting nodes in a pre-production environment with representative replicated tables and Distributed queries.
We verify that new replicas sync from healthy peers and that replication health is visible in system tables before the change reaches production.

How we hand over

You receive configuration files, runbooks for adding and removing nodes, and a record of the tests and their results.
We document the Keeper paths, access rules and rollback steps so your operators can apply the same procedure to future nodes.
Knowledge transfer sessions walk your platform team through the onboarding workflow and the failure cases we tested.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer, covering ClickHouse version changes that affect discovery.
Upgrade planning reviews the release notes for each ClickHouse version you adopt, since minor backward-incompatible changes can occur.
Confirm with us during scoping which support hours, response paths and review cadence the retainer includes before signing.

Automatic node registration with ClickHouse cluster discovery