ClickHouse Keeper health monitoring

Why Keeper health decides whether ClickHouse replicas stay in step

If you run replicated ClickHouse tables, Keeper health is the first thing to confirm before replicas fall behind. ClickHouse replicates data through ReplicatedMergeTree tables, and those tables depend on ClickHouse Keeper, the coordination service that replaces ZooKeeper, to agree on which parts exist and which replica should fetch them. The replication guide recommends at least three Keeper nodes, on dedicated hosts in production, with about 4 GB of RAM each. Check that your deployment meets that baseline before go-live, and confirm the exact sizing for your data volume with your own load tests, since the guide gives a starting point rather than a capacity figure.

A recovered replica syncs its missing data from a healthy replica without manual intervention, so a single node failure is rarely the operational risk. The more likely problem is a Keeper quorum or latency issue that stalls coordination without any visible outage. Queries keep returning results, inserts keep succeeding on some replicas, and the replication queue quietly grows on others. That gap is why Keeper needs its own monitoring and alerting, separate from checks on the ClickHouse servers that depend on it.

Plan this work before production go-live, after any Keeper incident, and when you move from another coordination setup. In each case, establish a baseline for Keeper responsiveness and quorum state, then decide who receives alerts and what action follows. Ask your implementation partner or internal team to confirm which Keeper metrics your ClickHouse version exposes, because the commands and fields you rely on depend on the release you run. The ClickHouse replication guide is the natural next read for the topology decisions behind these checks.

Which Keeper check fits which question

This table compares the Keeper four-letter commands ruok, mntr and lgif with the operator's own log and metric collection. Read each row as a question you need answered, then pick the check that answers it without guessing at exact output fields. The main takeaway is that no single command covers liveness, latency, leadership and trends, so combine them.

Check	What it tells you	How it runs	When it fits
ruok	Whether the Keeper process answers at all, a liveness signal rather than a health score	Four-letter command sent to the client port, easy to wrap in a probe script	Load balancer or orchestrator liveness checks and quick manual confirmation
mntr	Latency, outstanding request and data size figures for one node	Four-letter command returning a metric listing; confirm field names for your version	Spot checks during an incident and building a simple scrape source
lgif	The view of the Keeper log and leader state held by one node	Four-letter command run per node; compare output across all members	Confirming which node leads and whether members agree after an election
Operator log collection	Election events, session errors, warnings about slow requests and snapshot activity	Keeper server logs shipped to your log platform	Explaining why a change happened and finding warnings that precede replication lag
External metrics scraping	Trends in the same figures over time, so drift is visible before it becomes an outage	Regular scrape of Keeper metrics into your monitoring stack, with alert rules you define	Trend charts, alerting and capacity planning across weeks rather than seconds
Command enablement	Whether the four-letter commands are allowed on your Keeper version and configuration	Set in Keeper configuration; confirm the allowed list in the ClickHouse documentation	Before writing any probe, so checks do not fail silently after an upgrade

How to set up Keeper health checks step by step

This sequence confirms your Keeper ensemble matches the replication guidance, exposes only the four-letter commands your team needs, establishes a baseline, and proves that alerts fire and replicas recover before you rely on them in production.

Confirm ensemble size and host layout
Start by checking the Keeper ensemble against the ClickHouse replication guide, which recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each. Record each node's host, zone or rack, and role in a short inventory so later checks can be matched to specific machines. Confirm with your platform team that no Keeper node shares a host with a ClickHouse data node. The step is complete when the inventory matches the guide and every node is listed with its location.
Enable and restrict four-letter commands
Enable the four-letter word commands your monitoring needs in the Keeper configuration, limiting the list to the ones you intend to use, such as ruok, mntr and lgif. Restrict network access to the Keeper client and monitoring ports so only your collectors and administrators can reach them. Confirm the setting after a configuration reload by sending ruok from an approved host and from an unapproved host, expecting a response only from the approved one. The step is complete when the allowed command list is documented and the access test behaves as intended.
Capture a manual baseline on every node
Run ruok and mntr against each Keeper node by hand and save the output with a timestamp. A healthy node answers imok to ruok. Note the values from mntr for outstanding requests, average and maximum latency, and the number of followers, and keep them as your reference point. Repeat the checks during a quiet period so the baseline reflects normal load. The step is complete when every node has a saved baseline and all nodes report imok.
Record leader, follower and snapshot sizes
Run lgif on each node to record its role, the last committed log index, and the number of entries, then confirm that exactly one node reports as leader and the others as followers. Note the snapshot and log directory sizes on each host with standard filesystem tools, since growth in these directories is a warning sign that is easy to miss. Save the results alongside the latency baseline. The step is complete when the cluster has one leader, all followers match on the committed index within normal lag, and the directory sizes are recorded.
Schedule collection and alert on quorum and latency
Set up a scheduled job or your existing Prometheus or monitoring agent to poll ruok, mntr and lgif at a fixed interval and store the results. Define alerts for a node that stops answering imok, for loss of quorum, for a leader change outside a planned window, for a rise in request latency above your baseline, and for snapshot or log directory growth. Confirm that the collector writes data by checking the most recent timestamp in your store for each node. The step is complete when every alert rule is in place and each node shows recent samples.
Test alerts with a controlled Keeper restart
In a pre-production environment with representative data, stop one Keeper node in a controlled way, then restart it. Confirm that the node-down and leader-change alerts fire, and that the cluster keeps quorum throughout. Check that a ReplicatedMergeTree table on a replica stays writable, then confirm that the recovered replica catches up with the healthy replica without manual intervention, as the replication guide describes. Compare the Keeper latency and log size during the event with your baseline. The step is complete when the alerts fired as expected, the replica resynchronized, and the results are documented for your runbook.

Keeper warnings that usually come before replication problems

Keeper problems rarely announce themselves as replication failures. They show up first as slow answers, unstable leadership or growing state, and replicas then fall behind or go read-only. The cards below cover the signals worth checking early, and what to look at next when each one appears.

Node stops answering ruok

The ruok four-letter command is a basic liveness probe, and a Keeper node that stops replying to it is the clearest early warning. Check whether the process is still running, whether the host is under memory or disk pressure, and whether the node can reach its peers on the Raft ports. Confirm that the remaining nodes still hold a quorum, because replicated tables depend on it. Alert on missed responses, not a single timeout.

Repeated leader changes

One leader election after a restart is normal, but a leader that keeps changing points to an unstable cluster. Use mntr and lgif to see how often leadership moves and which node holds it. Look for network jitter between Keeper nodes, CPU starvation, slow disks that delay log writes, or a node that keeps rejoining. Frequent elections stall writes to ReplicatedMergeTree tables, so treat a repeating pattern as an incident to investigate.

Rising request latency in mntr

The mntr output reports request latency statistics, and a steady upward trend is often the first sign of trouble. Compare current values with a baseline captured during normal load, because acceptable values depend on your hardware and network. Check disk latency on the Keeper data directory first, since Keeper persists its log before acknowledging writes. Also look at whether ingestion spikes are creating unusually many coordination operations.

Growing outstanding requests

Outstanding requests that accumulate instead of draining mean the cluster is processing work more slowly than it arrives. Watch the outstanding request count in mntr alongside latency, since the two usually rise together. Identify which replicated tables generate the most coordination traffic, and check whether replicas are busy merging or fetching parts. A queue that does not clear after load drops is a sign to reduce coordination pressure or add capacity before writes back up.

Snapshot and log sizes that keep climbing

Keeper stores its state as snapshots and a change log on disk, and both should stay within a predictable range. Sizes that keep growing without a matching rise in data suggest snapshots are not being cleaned up, or that a node is lagging and holding back log compaction. Check the Keeper data directories, disk free space and the snapshot and log settings in your configuration. Growing storage on a Keeper node can also end in a node failure, so set alerts before the disk fills.

Replicas read-only or delayed

The clearest replication symptom is a replica that reports read-only status or falls behind its peers. Query the replication state in system tables to see whether a replica has lost its Keeper session, has a growing queue of replication tasks, or has a delay compared with the others. A recovered replica syncs missing data from a healthy replica on its own, so check whether that sync is progressing or stuck. Persistent read-only status usually points back to Keeper connectivity or quorum.

How a Keeper problem turns into a replication problem

Replicas depend on Keeper for their replication log, so a Keeper problem becomes a replication problem when replicas can no longer record or fetch entries through it. In a ReplicatedMergeTree setup, coordination state such as the replication queue and part metadata lives in ClickHouse Keeper. When quorum is lost or request latency rises, a replica may still accept inserts and run local merges, because the data is on its own disks. What it cannot do reliably is publish those changes to the other replicas or learn what they have committed. The result is drift: replicas stop agreeing on which parts exist and what still needs to be fetched, and the gap widens while the symptoms stay quiet at the application layer.

The sync-on-recovery behavior in the replication guide applies only after Keeper and the affected replica are healthy again. A recovered replica then pulls missing data from a healthy peer without manual work, which is why the outage often looks resolved. The risk is in the window before that point. Confirm with your team which Keeper signals (session loss, quorum status, and rising latency) you will alert on, and pair them with replica-side checks in the system tables: lag and queue depth in system.replicas, and the read-only flag that shows a replica has lost its coordination session. Keeper health tells you the cause, while replica lag and read-only status tell you whether the effect has reached your data.

Security and operating controls for Keeper monitoring

Keeper monitoring creates its own attack surface and its own evidence trail, so the controls around it matter as much as the checks themselves. The cards below cover network exposure, output handling, scrape credentials, host placement, change testing and retained history.

Restrict Keeper ports to monitoring hosts

The Keeper client port and the four-letter command port should accept connections only from the Keeper peers, the ClickHouse servers and the specific monitoring hosts that poll them. Commands such as ruok, mntr and lgif are read-only, but they reveal cluster state to anyone who can reach the port. Confirm the exact port and allow-list configuration keys for your ClickHouse version, and test from an address outside the allow list to confirm the block works.

Treat command output as sensitive

Output from `mntr` and `lgif` describes your replication topology, including node roles, outstanding requests and session counts. Store scraped results in the same access-controlled systems as other infrastructure telemetry, and avoid pasting raw output into tickets or chat channels that include people outside the operations team. Ask your security team to classify this data alongside your other operational logs.

Protect the monitoring account

Whatever collects Keeper metrics or runs health checks needs a dedicated identity with the narrowest access that works. Store its scrape credentials in your secrets manager, rotate them on a schedule your team can actually follow, and remove them from exporter configuration files that are readable by other users on the host. Confirm which authentication options your Keeper and exporter versions support before you design the rotation process.

Keep Keeper on dedicated hosts

The replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each. Running Keeper on the same machines as heavy merge or query workloads makes its latency less predictable, which is the signal you most need to trust. When you review a design, check that Keeper nodes do not share hosts with ClickHouse data nodes, and that their disks are sized for snapshot and log growth.

Test Keeper changes before production

The production FAQ recommends a pre-production environment with representative data and queries, and reading the changelog before upgrading because minor backward-incompatible changes can occur. Apply the same rule to Keeper version changes and to changes in monitoring thresholds or scrape settings. Run the new Keeper version against a copy of your topology, confirm that ruok, mntr and lgif still return the values your dashboards expect, and only then roll it to production one node at a time.

Retain Keeper health logs and alerts

Keep Keeper server logs, the metrics history behind your dashboards and the history of fired alerts for a period your incident review and compliance obligations require. Without this record, a replication incident that started with leader churn or rising latency is hard to reconstruct after the fact. Confirm with your team how long each source is kept, where it is stored and whether it is included in your backup scope.

Questions teams ask before they monitor Keeper themselves

Are ruok, mntr and lgif enough to monitor ClickHouse Keeper without a metrics stack?

They are enough to answer a point-in-time question, but not enough to catch a trend. The ruok command reports whether a node is serving, mntr returns counters such as outstanding requests, latency and znode counts, and lgif shows log and commit indexes that reveal a node falling behind. Each one is a snapshot. A leader change, a slow fsync or a growing snapshot shows up only if someone runs the command at the right moment. Teams that run ClickHouse for anything they cannot rebuild usually scrape these values on an interval and alert on changes, which needs a collector, a store and an alert route. Confirm the exact fields your Keeper version returns before you write alert rules against them.

How much effort do the initial setup and alerting take, and what drives it?

The effort depends on a few inputs more than on Keeper itself. The main drivers are how many ClickHouse clusters share the Keeper ensemble, whether you already run a metrics and alerting stack such as Prometheus, Grafana or Zabbix, how alerts are routed to people who can act on them, and how much access your team has to the Keeper hosts. A single cluster with existing tooling is a small job. Several clusters, new tooling, and a routing model that reaches an on-call rota take longer. Ask any provider to break the quote into collection, dashboards, alert rules, runbooks and handover, and to state which parts are one-time and which recur as Keeper versions change.

Can an in-house team set up Keeper monitoring without outside help?

Yes, if the team already operates ClickHouse and has someone who understands Keeper's quorum behavior. The checks themselves are simple to script. The harder parts are choosing thresholds that separate noise from a real replication risk, testing that alerts fire, and writing runbooks that tell the on-call engineer what to do with a leader change at 3 a.m. Teams without that experience often get the collection working and then struggle with alert quality. An outside partner can fill that gap through a fixed-scope engagement, where scope and deliverables are agreed in writing before work starts and the price follows from that written scope.

What does the work need from our team?

Keeper monitoring needs more from your side than from the tooling. You will need host access to each Keeper node so checks can run and collectors can be installed, a pre-production environment with a representative ensemble where alert rules can be tested without paging anyone, a named on-call owner who accepts the alerts, and agreed escalation paths that say who acts on a quorum warning and who is told about a replication lag warning. Without these, alerts tend to fire into a channel no one reads. Confirm early whether your security team permits the four-letter commands on the client port and from which networks, because that decision shapes the whole design.

How does Keeper monitoring relate to backups and upgrades?

Keeper monitoring tells you whether the coordination layer is healthy before you take a backup or start an upgrade. Replication alone does not protect against human error or corruption, so backups with BACKUP and RESTORE still matter, and the backup documentation recommends regular restore tests on spare clusters. Keeper health affects both: an ensemble that is already losing quorum or building large snapshots is a poor time to change versions or run a heavy backup. Use Keeper metrics as a gate in your upgrade runbook, and confirm the version support window in the ClickHouse security policy before you pick a target release.

Which Keeper warnings should we treat as early signs of replication trouble?

Watch for repeated leader changes, rising request latency, growing outstanding request counts, and snapshot sizes that keep increasing between checks. Each one can precede replicas falling behind, because ReplicatedMergeTree tables depend on Keeper to coordinate. The ClickHouse replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each, so check whether your hosts meet that baseline before you read the metrics as a sizing problem. A recovered replica syncs missing data from a healthy replica on its own, which means the warnings you need to act on are the ones that show Keeper itself losing its footing.

Limits and items to confirm in your environment

The published ClickHouse resources describe the Keeper pattern, not your hardware, network or release line. The cards below list what to confirm before you rely on any command, threshold or alert from this page, since each one changes what healthy looks like for your cluster.

Four-letter command support

Confirm which four-letter commands your Keeper build answers, such as ruok, mntr and lgif, and which fields each one returns in your version. Field names and availability can differ between releases, so check output on a test node against the vendor documentation for your build. Keep a saved sample of each command's output from a healthy period so later comparisons are against your own baseline, not a field list you remember.

Standalone or embedded Keeper

Establish whether your Keeper runs as a standalone ensemble on dedicated hosts or is embedded in the ClickHouse server process. The commands, ports, log locations and failure domains differ between the two, and an embedded setup means a server restart can also affect coordination. Write down the topology in your runbook, including which hosts hold which role, before you choose where to scrape or probe.

Supported ClickHouse release line

Check your running version against the supported list in the security policy before you set alert thresholds, since monitoring tuned for an unsupported release may chase metrics that have changed. Stable releases ship roughly monthly, and LTS releases ship twice a year, so confirm which line you follow and when you plan to move. Read the changelog before any upgrade, because minor backward-incompatible changes can occur.

Scrape and call access

Verify that your monitoring stack can reach the Keeper client port from the hosts that run your checks, and that firewall and network policy allow it. Some stacks scrape over HTTP, while others need a script that opens a TCP connection and sends the command text, so confirm which method your tooling supports. Also confirm who may send these commands, because a Keeper that answers mntr to any host can leak cluster details.

Keeper log and snapshot retention

Confirm how long Keeper logs and snapshots are kept, where they are stored, and whether rotation or cleanup is handled by your configuration or by an external job. Snapshot sizes and log volume grow with cluster activity, so a disk filling quietly can stop Keeper before any command reports a problem. Set disk alerts on those paths and confirm that retention still covers the window you need to investigate a replication incident.

Thresholds for your workload

Derive latency, leader-change and snapshot-size thresholds from your own cluster's normal behavior over a representative period, not from a generic number. Hardware, network round-trip time, insert rate and replica count all shift what is normal. The published guidance describes the pattern and recommended sizing, but it does not provide baselines for your hardware, so measure first, then alert on deviation and sustained change.

How DeployOpen scopes and supports Keeper monitoring

DeployOpen plans, builds and hands over Keeper monitoring inside your own infrastructure, so the health checks, alert thresholds and runbooks match how your ClickHouse replicas are actually deployed. Each engagement starts from a written scope, and ongoing support is available as a retainer once the monitoring is in place.

How we scope

We review your Keeper node count, host placement and whether each node runs on dedicated hardware.
We agree in writing which four-letter commands, metrics and log sources the monitoring will cover.
We confirm which alert thresholds, notification channels and on-call owners the written scope includes.

How we build

We enable and verify the Keeper client ports and the commands your monitoring tooling will call.
We wire leader changes, latency, outstanding requests and snapshot size into your existing metrics stack.
We test each alert against induced conditions in a pre-production cluster before it reaches production.

How we hand over

We document each check, its expected healthy value, and the replication symptoms it is meant to predict.
We provide runbooks that name the first diagnostic steps for each Keeper warning your team will receive.
We walk your operators through the dashboards and alert routing so they can run the checks without us.

How we support

Ongoing support is available as a retainer for threshold tuning, version changes and new Keeper nodes.
Retainer scope, response expectations and review cadence are agreed in writing before the retainer starts.
Monitoring changes follow your change control, and we review them against the Keeper release notes you use.