Why ClickHouse Keeper security decides whether replicated tables stay safe
Securing ClickHouse Keeper protects the coordination layer that tells each replica what data it should hold, which parts it must fetch and how merges are agreed across nodes. If your self-managed cluster uses ReplicatedMergeTree tables, that metadata is only as trustworthy as the Keeper ensemble behind it. Teams usually need this work before go-live, after an audit finding that flags the coordination service, or when Keeper hosts share machines with other services that were never meant to sit in the same trust zone. Whichever trigger applies, the time to decide the controls is before the first production table is created.
ClickHouse Keeper replaces ZooKeeper as the coordination service for replication in ClickHouse. The replication guide recommends at least three Keeper nodes, on dedicated hosts in production, with about 4 GB of RAM each. Those recommendations shape the security boundary: a dedicated ensemble can be isolated, firewalled and monitored as its own asset, while a Keeper process co-located with query-serving workloads inherits their exposure. Ask your team to confirm the node count, host placement and memory allocation against the current replication documentation before you finalize the design, since sizing guidance can change between releases.
Keeper's role also explains why its compromise is serious. Replicas trust coordination state to decide what to replicate, so unauthorized write access to that state can cause divergence or data loss that replication alone will not catch. Treat Keeper as production infrastructure with its own access review, change control and restore procedure, and plan for the question an auditor will ask: who can reach the coordination service, and how would you know if that changed? The sections that follow cover the specific controls for doing so.
Which Keeper controls fit which deployment
This table compares five Keeper controls by what each protects, when it fits and the cost it adds to operations. Read each row against your topology: a single-site cluster, a multi-team shared cluster and a regulated environment with audit obligations usually need different combinations. The main takeaway is that every control adds a management task, and that task has to be owned before it is switched on.
| Control | What it protects | When it fits | Trade-off or check needed |
|---|---|---|---|
| Digest ACLs on replicated table paths | Keeper znodes for ReplicatedMergeTree tables from unauthorized reads or writes by other clients sharing the ensemble | Shared Keeper ensembles where more than one team or application creates replicated tables | Every replica that shares a path must use the same digest credentials, so credential changes must be rolled out consistently |
| TLS for client connections from ClickHouse servers | Coordination traffic between ClickHouse servers and Keeper against interception or tampering on the network | Keeper nodes reached across hosts, racks or any network segment that is not fully trusted | Adds certificate issuance and rotation work on both Keeper and server configurations; confirm the rotation procedure before go-live |
| TLS for Raft traffic between Keeper nodes | Consensus messages exchanged between Keeper members from eavesdropping or spoofed peers | Keeper members spread across hosts or sites, or any environment with a policy requiring encryption in transit | Certificates must be valid on every member at the same time, so a rotation that misses one node can break quorum; test it on a staging ensemble |
| Four-letter-word command allow list | Keeper's diagnostic commands, which can expose cluster state and operational detail to anyone who can connect | Every production ensemble, with the list limited to the commands monitoring and runbooks actually use | Monitoring tools and health checks break if a command they depend on is omitted; list the commands each consumer needs before restricting |
| Isolating Keeper on dedicated hosts or a separate network segment | Coordination service from resource contention with ClickHouse servers and from access by hosts that do not need it | Production clusters where Keeper must stay available under load or where only ClickHouse servers should reach it | Adds hosts and firewall rules to maintain; the reference guidance calls for dedicated hosts in production, so confirm capacity and sizing with your team |
Four exposure risks to close before go-live
Keeper holds the coordination state that ReplicatedMergeTree tables depend on, so a weakness there can affect every replica that relies on it. The four risks below cover who can read that state, who can observe the traffic, who can issue administrative commands, and what else shares the hosts.
Unauthenticated access to table metadata
Replicated tables store their metadata, replica state and part information in Keeper paths. If clients can reach Keeper without authentication, anyone on that network path can read or alter the znodes that describe your replicated tables. Control: enable Keeper authentication and set digest ACLs on the paths ClickHouse uses for replicated tables, so only the ClickHouse servers that hold the credential can read or change them. Check that the credential is not stored in plain SQL or shared configuration files that other teams can read.
Plaintext coordination traffic
Keeper client sessions and Raft traffic between Keeper nodes carry coordination state in clear text when TLS is not configured. On a shared network, anyone able to capture packets can observe replica membership and metadata changes, and an attacker who can inject traffic may disrupt consensus. Control: configure TLS for both client connections from ClickHouse servers and the inter-node Raft channel, and disable the plaintext listener on any port that is reachable from outside the cluster. Verify with a packet capture on a test network that no coordination payloads are readable.
Administrative four-letter commands
Keeper accepts four-letter-word commands such as status and configuration queries on its client port, and some of them reveal topology and state. When the allow list is left broad, any host that reaches the port can run them, including hosts that have no reason to manage the cluster. Control: restrict the four-letter-word allow list to the specific commands your monitoring and runbooks need, and limit which source addresses can reach the client port. Confirm the exact command names and defaults against the version you deploy.
Keeper co-located with busy workloads
When Keeper shares a host with ClickHouse query nodes, a heavy merge, a large query or an unrelated service can starve it of CPU, memory or disk I/O. Slow Keeper responses can then delay replication, cause session expirations and trigger replica failover at the worst moment. Control: run Keeper on dedicated hosts in production, with at least three nodes and enough memory for the Keeper process, and keep those hosts out of the ClickHouse query path. Ask your platform team to confirm the sizing guidance for your release before procurement.
Securing ClickHouse Keeper step by step
This sequence moves Keeper from a default coordination setup to a locked-down layer: scoped to the replicated tables it serves, isolated on its own hosts, encrypted on the wire, authenticated per table path, and exposing only the diagnostics monitoring needs. Each step closes with a check you can repeat after every change.
Inventory Keeper nodes, tables and clients
Start by listing every Keeper node, every ReplicatedMergeTree table and database that depends on it, and every host that connects to Keeper: ClickHouse servers, monitoring agents, backup jobs and administrator workstations. Record the Keeper ports in use and which hosts currently reach them. This inventory is the baseline for every later rule. You are done when each replicated table path maps to the servers that write to it and each Keeper client has a named owner. Confirm the current Keeper configuration keys for your version in the ClickHouse replication documentation before you change anything.
Place Keeper on dedicated hosts
The replication guide recommends at least three Keeper nodes, on dedicated hosts in production, with about 4 GB of RAM each. Move Keeper off the ClickHouse data nodes if it shares them, so that a heavy merge or query workload cannot starve coordination and so that a compromise of a data node does not automatically give an attacker a neighbor on the coordination layer. Ask your infrastructure team to confirm host placement across failure domains. You are done when every Keeper node runs on its own host, the ensemble has an odd number of members, and a single host loss leaves a working quorum in a test.
Restrict network access to Keeper
Create firewall or security group rules that allow Keeper client and Raft traffic only from ClickHouse servers and from named administrative hosts, and allow monitoring only on the ports it needs. Block all other sources, including application subnets that have no reason to reach coordination. Keep the Raft peer ports open only between Keeper nodes. Verify from an unapproved host that the client port and the Raft ports do not respond, and from an approved ClickHouse server that the client port does. Record the rule set in version control so drift is visible.
Enable TLS for client and Raft traffic
Enable TLS on the Keeper client port and on the Raft inter-node communication, using certificates issued by your internal certificate authority with clear server and client identities. Decide how certificates are issued, how long they live, and how rotation happens without a full ensemble restart, and document the owner of each step. Confirm the TLS settings for your ClickHouse version in the documentation before editing files. You are done when a connection without a valid client certificate fails, the ensemble re-establishes quorum after a rolling restart, and a test rotation completes on a non-production ensemble.
Configure digest ACLs for replicated table paths
Set digest-based ACLs on the Keeper znodes that hold replicated table metadata so that only the ClickHouse servers with the matching credentials can read or modify those paths. Scope each ACL to the table or database subtree it protects, and keep credentials in a secrets store rather than in plain configuration files shared across teams. Afterward, restart one replica and run an insert and a read through the replicated table. You are done when the replica rejoins, the expected data replicates to its peer, and an unauthorized client cannot read the protected paths.
Restrict the four-letter-word allow list
Keeper accepts four-letter administrative commands over its client port. Limit the allow list to the specific commands your monitoring and health checks need, and remove the rest. Confirm which commands are in the allow list for your version in the Keeper documentation, and check that your monitoring still gets the health and status data it depends on. You are done when a command outside the allow list returns no useful output from an approved host, and your health checks and dashboards still report each node's role and status.
Verifying that Keeper rejects what it should
Each control in this section needs a test that would fail if the control were missing. Start with the digest ACLs on replicated table paths. From a client that holds no Keeper credentials, attempt to list, read and write under the ZooKeeper path used by a ReplicatedMergeTree table, and confirm that every attempt is refused. Then repeat the read with a wrong digest credential, since a refusal for the wrong reason can hide a misconfigured ACL. Also confirm that a legitimate ClickHouse server can still read and write the same path, because an overly tight ACL will stop replication just as surely as a loose one lets strangers in.
Next, test the transport and the network boundary. From a host inside the cluster network, open a plaintext connection to the Keeper client port and to the Raft port, and confirm that each is refused wherever you have required TLS. Then send the four-letter commands, such as ruok and mntr, from a host that is not on the allow list and from a host that is. The untrusted host should get no useful reply, and the monitoring host should get the expected response. Finally, confirm from outside the approved server and monitoring subnets that Keeper host ports do not answer at all. Verify these settings against the Keeper documentation for your ClickHouse version, since the exact configuration keys and behavior should be confirmed before you rely on them.
Last, test recovery, because a Keeper control that blocks access can also block a replica from catching up. Stop one replica, insert rows into a replicated table on a healthy peer, restart the stopped replica, and confirm that its row counts converge and its replication queue drains without manual steps. The ClickHouse production FAQ advises testing changes on a pre-production environment with representative data and queries, so run this whole set there first, using tables that match your production schema and volumes. Record each result with the change request so that an auditor or on-call engineer can see what was tested and when.
What to confirm for your ClickHouse and Keeper versions
The controls on this page depend on which ClickHouse and Keeper builds you run, how your certificates are issued, and how your monitoring talks to the coordination layer. The cards below list the items to verify against vendor documentation and a test cluster before you change anything in production.
Confirm your version is supported
Check the exact ClickHouse and Keeper build in production against the versions the security policy lists as receiving security updates. Any other version is unsupported, so a Keeper hardening plan built on it will not receive fixes for issues found later. Record the full build string from each node, not only the major line, because support is tracked per listed version. If you are behind, plan the upgrade before the hardening work, since TLS and ACL behavior can differ between releases.
Check TLS and ACL options in your build
Before writing Keeper configuration, confirm which TLS settings and digest ACL options your exact version accepts. Option names and defaults change between releases, and an unrecognized key can stop a node from starting. Test the configuration on a non-production Keeper quorum first, and confirm that replicated table paths still resolve after ACLs are applied. Ask your vendor or check the replication documentation for the supported authentication mechanisms in that build.
Plan certificate rotation for a live quorum
Find out whether your version reloads Keeper certificates without a restart, and whether the Raft peers keep communicating while nodes are rotated one at a time. A rotation that breaks quorum membership can stall writes to every ReplicatedMergeTree table that depends on it. Test the full rotation sequence on a spare cluster, including trust bundle changes, expiry alerts, and rollback. Document which node you rotate first and how you confirm the quorum has a leader before moving to the next.
Verify monitoring against the allow list
Determine which four-letter commands your monitoring agents, health checks and load balancers send to Keeper, and compare that list with the allow list you intend to set. A probe that uses a command outside the allow list will fail silently or raise false alarms, which can hide a real outage. Confirm the exact command names your Keeper build supports, then add the allowed set to your monitoring configuration in the same change as the allow list itself.
Size memory and hosts for your node count
The replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each. Confirm that your planned hosts meet that guidance, that they are isolated from ClickHouse data nodes, and that your node count still gives a majority after one node fails. Ask whether your Keeper snapshot and log volume growth needs more disk than the default layout provides, and test memory use under your real replication load, since figures from documentation describe a starting point rather than a guarantee.
Read the changelog before each upgrade
Keeper behavior can change between stable and LTS lines, and minor releases can include backward-incompatible changes. Read the changelog for every version between your current build and the target, and list any changed defaults, removed settings or new ACL behavior. Run the upgrade on a pre-production cluster with representative data and queries, then confirm that replicas resync after a Keeper node restarts. Keep the previous package available so you can roll back a single Keeper node if the new version misbehaves.
What buyers ask about the cost and effort of securing Keeper
What actually drives the effort of securing ClickHouse Keeper?
The main drivers are the number of Keeper nodes, how many replicated tables depend on the ensemble, whether your organization already runs a certificate authority that can issue server and client certificates, and whether Keeper already sits on dedicated hosts. A three-node ensemble on dedicated hosts with an existing internal CA is a much smaller job than a mixed deployment where Keeper shares hosts with ClickHouse servers and certificates are issued by hand. Each replicated table also adds ACL work, because every table path in Keeper needs a decision about who may read and write it. The ClickHouse replication guide recommends at least three Keeper nodes on dedicated hosts in production, so ask your team to confirm the current topology before estimating anything.
What part of Keeper security is one-time work and what recurs?
The one-time work covers design, configuration and verification: choosing the digest ACL scheme for replicated table paths, enabling TLS for client and Raft traffic, setting the four-letter-word allow list, isolating the Keeper hosts, and proving each control with a test. The recurring work covers certificate rotation, version upgrades that must be checked against the security configuration, and ACL changes whenever new replicated tables or databases are added. Recurring work is usually small per cycle but easy to forget, so ask for a runbook that names who rotates certificates and who reviews Keeper ACLs after each schema change.
What does our team need to provide before the work can start?
Expect to provide a test cluster that mirrors the production Keeper topology, access to the certificate authority or its issuing process, approval for the network changes that restrict Keeper ports, and a scheduled maintenance window for rolling restarts. Keeper ensembles change their configuration node by node, so the window must allow for rolling restarts and a rollback plan. You should also name an owner who can approve ACL changes for each replicated table path. Ask any partner to list these prerequisites in the written scope so that gaps are visible before the schedule is fixed.
Can our internal team secure Keeper without an outside partner?
Yes, a team can do this alone if it has Linux administration skills, working knowledge of TLS certificates and their trust chains, hands-on ClickHouse operations experience, and a test environment where changes can be tried before production. The work is mostly careful configuration and verification rather than new architecture. Teams without a test environment, or without anyone who has restarted a Keeper ensemble under load, tend to take more risk when they change live coordination nodes. A partner is most useful when the scope is fixed in writing, the team lacks one of those skills, or the work must be completed alongside other hardening tasks.
How long will securing Keeper take in our environment?
The duration depends on the environment, so no general figure applies. It depends on the topology, the state of your certificate infrastructure, how many replicated tables need ACLs, how many change approvals your network process requires, and how much testing your maintenance windows allow. Ask for a fixed-scope estimate based on a written scope that lists the nodes, the tables, the certificate process and the verification steps. A written scope also makes it clear which items are included, which need your input, and which fall outside the engagement.
What does a fixed-scope engagement for Keeper security cover?
A fixed-scope engagement is agreed in writing before work starts and typically lists the Keeper topology in scope, the controls to apply, the verification tests, the documentation to hand over and the client inputs required. For Keeper, that usually means digest ACL design for replicated table paths, TLS configuration for client and Raft traffic, the four-letter-word allow list, host isolation and a runbook for rotation and upgrades. DeployOpen plans, implements, hardens and documents self-hosted platforms in your own infrastructure. It does not issue audit reports or certifications, so any compliance assessment remains with an independent auditor.
What should we ask for in a quote for Keeper hardening?
Ask the quote to show each deliverable separately, including design, configuration, testing and documentation, so you can see which parts are one-time and which recur. Request a list of client prerequisites, the number of Keeper nodes and replicated tables in scope, the verification method for each control, and what happens if a test fails. Ask whether certificate rotation and ACL review after new tables are included, or whether they require a separate retainer. Retainers are available for ongoing support, upgrades and testing, so a quote should state clearly which part is a project and which part continues after handover.
What do Keeper hardening upgrades and rotations cost in effort over time?
The ongoing effort follows three triggers: certificate expiry and rotation, ClickHouse version changes, and new replicated tables. Each rotation requires a rolling restart of Keeper nodes and a check that clients still connect over TLS. Each upgrade requires confirming that the security settings still apply, and reading the changelog, since the ClickHouse production FAQ notes that minor backward-incompatible changes can occur. Each new table requires an ACL decision. Ask your engineers to estimate how often these events occur in your environment, because that frequency determines whether a retainer or an internal runbook is the better fit.
How DeployOpen works on this
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For Securing ClickHouse Keeper, we start from your replicated tables, Keeper topology and network layout, then agree a fixed scope in writing before any change is made to your coordination layer.
How we scope
- We inventory which ReplicatedMergeTree tables and Keeper nodes the written scope covers in your environment
- We agree in writing whether digest ACLs, TLS for client and Raft traffic, and the four-letter-word allow list are in scope
- We confirm which hosts are dedicated Keeper nodes, and document any co-located workloads that need to move or be accepted as risk
How we build
- We apply Keeper configuration changes in a pre-production cluster with representative replicated tables before production
- We test that replicas still sync and that clients without the right credentials or certificates are refused
- We verify the allow list and port exposure from the network paths your monitoring and clients actually use
How we hand over
- We deliver the configuration, ACL and certificate inventory, and the change record for each Keeper host
- We document the rollback steps and the restore checks we ran, so your team can repeat them without us
- We walk your operators through the Keeper health checks and the alerts to keep after handover
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the original engagement
- Certificate rotation and ACL changes can be handled under a retainer when your team asks us to run them
- Our work prepares evidence for compliance assessments, but independent auditors and certification bodies issue any audit outcome
