How to upgrade ClickHouse Keeper without losing quorum
Your cluster runs ReplicatedMergeTree tables, and those tables depend on ClickHouse Keeper for coordination. You are planning to move the servers to a target version on one of the supported lines: 26.9, 26.8, 26.7 or 26.3, according to the ClickHouse security policy. Keeper can be upgraded as its own step, separate from the server rollout, but the ensemble must keep a quorum of voting members at every moment of the restart. Treat the supported server upgrade path as your starting point, then plan the Keeper step around it rather than assuming the two can be handled in one pass. Before scheduling anything, read the release notes for the target line and look specifically for Keeper-specific guidance, since a change to coordination behavior would alter the order of operations.
Two questions need answers before you touch a node. First, confirm in the ClickHouse documentation whether Keeper and server versions must match or may differ during a transition, and whether a mixed-version ensemble is supported and for how long. Second, confirm the current ensemble size and the health of each member, since a three-node ensemble tolerates one node down and any second failure removes quorum. The replication guide recommends at least three Keeper nodes on dedicated hosts in production, with about 4 GB of RAM each, so check that your hosts meet that baseline before adding load from a restart. Keep a record of the current Keeper configuration and the data directories on every node, because you will compare them afterward.
Restart Keeper one member at a time, wait until the restarted node rejoins and reports a healthy role, then move to the next. Stop after any step that does not recover and resolve it before continuing, because a second node down in a three-node ensemble means the coordination layer stops accepting writes. Take a backup of the Keeper data directories before the first restart so that you can restore a member if a snapshot or log file does not load on the new version. After the last node is back, verify the ensemble by checking that a single leader exists, that all members report as followers or the leader with matching state, and that replicated tables accept inserts and sync their parts. The exact commands and status fields should be taken from the ClickHouse documentation for your target version, not from memory.
What changes in the releases that drive a Keeper upgrade
This table compares the release signals that can force a Keeper upgrade, the impact each one has on a running ensemble, and the action to take before scheduling work. Read it as a checklist: release cadence sets how often you must decide, while changelogs and Keeper-specific notes decide whether a given version is safe for your nodes.
| Release signal | Impact on Keeper | Action needed |
|---|---|---|
| Monthly stable releases | Stable lines ship roughly monthly, so a Keeper version can fall out of step with server releases within a few months. | Decide on a cadence for reviewing new stable releases and record which server and Keeper versions run together. |
| Backports for the three latest stable lines | Bug fixes and diagnostics reach only the three latest stable lines, so an older Keeper line may stop receiving them. | Confirm which Keeper line you run is still in the backport window before choosing a target version. |
| LTS releases twice a year | LTS lines are supported for one year after release, which sets the window for planning a longer-lived Keeper version. | Map your Keeper line to its end of support date and plan the next move before that date arrives. |
| Current stable v26.9.20.23-stable and LTS lines v26.8.29.10-lts and v26.3.53.4-lts (published October 16, 2026) | These are the release lines listed on the GitHub releases page at the time of publication, and they set the candidate targets for a Keeper move. | Check the current release list before the change window, because newer builds may have been published since this page was written. |
| Minor backward-incompatible changes | The production guidance warns that minor releases can contain backward-incompatible changes, which may affect Keeper behavior as well as server behavior. | Read the changelog for every version in the path, then test the target on representative data before production. |
| Keeper replaces ZooKeeper | Keeper is the coordination service behind ReplicatedMergeTree, so a Keeper upgrade changes the component that decides replica state and quorum. | Treat Keeper as its own change with its own rollback and verification steps, not as a side effect of a server upgrade. |
| Keeper release notes versus server notes | Server changelogs may not describe Keeper changes in the same detail, so a server release note alone does not confirm Keeper compatibility. | Confirm Keeper release notes separately from server notes for each target version and record the findings in the change plan. |
Why a three-node Keeper ensemble can survive one restart
The replication guide recommends at least three ClickHouse Keeper nodes in production, on dedicated hosts with about 4 GB of RAM each. The reason is arithmetic: a three-node ensemble keeps quorum when one node is down, so a rolling Keeper restart can take one node out at a time, but only if the other two are fully healthy. Restart the next node only after the ensemble reports a leader and both remaining followers are caught up. Check this on each node before moving on, using Keeper's four-letter status commands and the replica health views on your servers, and confirm the exact commands for your Keeper version in the replication guide before you script them.
The risk is a slow recovery. A restarted node must rejoin, replay its log and fetch a snapshot from the leader if it has fallen behind. While that node is still catching up, it does not count toward quorum, so a second restart at that moment leaves the ensemble with a single voting member and no ability to write. Build the runbook around a hard gate: no restart begins until the previous node has rejoined and the ensemble has been stable for a period you agree on in advance. Record the time each node takes to recover during the first run in a pre-production environment, since that measured value, not a number from a document, is what should set the gap between restarts in production.
Dedicated hosts matter because Keeper holds its state in memory and is latency sensitive. On a shared host, server merges and queries can consume memory and I/O that Keeper needs to answer heartbeats and apply writes, and a stalled Keeper node looks to the rest of the cluster like a failed one. Keeping Keeper on its own hosts, sized at or above the RAM figure in the guide, removes that competition. Sizing beyond these facts depends on your own load, including the number of replicated tables, insert rate and znode count, so ask for a load test against a representative workload and confirm the headroom before you decide how many nodes to restart in a window.
What to check before touching any Keeper node
Keeper changes are hard to reverse once replicated tables depend on the new ensemble, so the checks below come before any restart. Each card covers one item the team should confirm in writing: a backup that can be restored, a tested restore, a version and configuration inventory, a rehearsal on representative data, a health baseline, and a named decision owner.
Verified backup of replicated tables
Take a BACKUP of every replicated table before the first Keeper node is touched. The backup documentation is explicit that replication alone does not protect against human error or corruption, so a healthy replica copying bad data would copy the damage to the rest of the cluster. Confirm the backup target (local disk, S3-compatible storage or Azure Blob Storage), confirm it is complete for every database and table in scope, and record where the backup set is stored and who can read it.
Restore test on a spare cluster
A backup that has never been restored will probably not restore properly when it is needed. Restore the backup set to a spare cluster built from the same ClickHouse and Keeper versions that production runs, then run row counts and a sample of business queries against the restored tables. Record the elapsed time of the restore, because that figure is the realistic floor for any rollback decision made during the change window.
Keeper configuration and version inventory
Write down the Keeper version, the ClickHouse server versions that connect to it, and how each Keeper node was installed (Linux package, container image or Kubernetes manifest). Capture the Keeper configuration file for every node and compare the nodes for drift, such as differing server IDs, ports or storage paths. A mismatch found here is far cheaper to fix before the restart than during it.
Pre-production copy with representative data
The production FAQ recommends a pre-production environment with representative data and queries before upgrading. Build a copy of the replicated cluster, including a Keeper ensemble of the same size, and perform the Keeper restart sequence there first. Confirm that replicated tables stay writable and that a restarted node rejoins the ensemble, and keep the timings and any errors so the production runbook matches what was actually observed.
Ensemble health and replica sync baseline
Before the change, record the state of the ensemble and of every replicated table: which node is leader, whether all Keeper nodes are reporting, replication queue lengths, and whether any replica is lagging or read-only. A recovered replica automatically syncs missing data from a healthy replica, but that only helps if the baseline shows which replicas were healthy to begin with. Save these outputs so post-upgrade checks have something exact to compare against.
Named owner and rollback decision point
Name one person who owns the Keeper change, the change window and the decision to stop or roll back. Agree in writing which conditions trigger a stop, such as lost quorum, a replica that cannot sync or query errors on replicated tables, and who has authority to make the call. The window should be long enough for the rollback path to finish, which is why the restore test timing matters for this decision.
Rolling Keeper restart, one node at a time
This sequence upgrades each Keeper node in turn while the ensemble keeps quorum and replicas stay available, so the Keeper layer moves to the target version without a cluster-wide outage. Each step ends with a check that tells you whether to continue or stop.
Confirm quorum and replica health before starting
Before touching any Keeper node, confirm the ensemble has a leader and a working quorum, and that every ReplicatedMergeTree table reports its replicas as in sync. Run these checks from the Keeper and ClickHouse monitoring you already have, and record the results. If the ensemble is already degraded or a replica is lagging, stop here: the rollback point is simply not starting. Do not begin the restart sequence until the baseline is clean and written down.
Stage the target Keeper package in pre-production
Install the target Keeper version on a pre-production ensemble with representative data and run the same rolling restart there first. Your platform or database team performs this, with DeployOpen if you have engaged us for the work. Confirm the ensemble reforms, the leader is elected, and replicas resume normal replication. The rollback point is the pre-production run: if it fails, production stays on the current version. Check the release notes and the Keeper documentation for the target version for any configuration changes that must be made before the upgrade.
Upgrade and restart one follower node
Choose a follower, not the current leader, and stop its Keeper process. Install the target package, apply any configuration changes listed for the target version, and start the node. The exact commands, configuration keys, and how snapshot and log files are read or converted must come from the official Keeper documentation for your target version, not from this page. Keep the old package available so you can reinstall it if the node will not start. The rollback point is reinstalling the previous package on this one node while the rest of the ensemble still holds quorum.
Verify the upgraded node rejoins the ensemble
Watch the restarted node until it reports as a follower with the current leader and appears in the membership list. Check the Keeper logs for snapshot loading or log replay errors, and confirm the node's version reads as the target version. The step is complete when the ensemble shows a single leader and all configured members are healthy. If the node cannot rejoin, stop the sequence, keep the remaining nodes on the old version, and investigate before continuing.
Let replicas settle before the next node
Wait for ClickHouse replicas to finish any queued replication work after the Keeper change. A recovered replica automatically syncs missing data from a healthy replica without manual intervention, so you are looking for the replication queues to drain and the replica status to return to normal, not for manual repair. Check replication lag and error counts on the ReplicatedMergeTree tables. Continue only when the replicas report in sync and no new errors appear in the server logs.
Repeat for each remaining node
Apply the same cycle to every remaining Keeper node, one at a time, with the leader last. After each node, rerun the quorum and replica checks from the first step. Keep a written record of the version and health result for each node so you can show the order and state of the rolling restart later. If any node fails a check, stop, and do not upgrade another node until quorum and replica health are restored.
Return to the server upgrade order
Once every Keeper node runs the target version and the ensemble is healthy, move to the ClickHouse server upgrade. Follow the server upgrade order defined in the release notes for your target version, and do not assume the Keeper and server steps can be reversed or interleaved. Confirm the Keeper ensemble is still healthy at the start of the server work. The sequence is complete when Keeper and server versions match the plan and the post-upgrade verification for the ensemble has been recorded.
What makes a Keeper upgrade take longer
The table lists the factors that lengthen or shorten a Keeper upgrade and whether each one is paid once or repeats at every release. Read the direction column first, then use the recurrence column to decide which drivers to fix in the initial design and which to re-check each time you move to a new version. The main takeaway is that node count, host sharing and monitoring coverage decide most of the schedule.
| Driver | Effect on effort | One-time or recurring |
|---|---|---|
| Number of Keeper nodes | More nodes mean more sequential restarts and more quorum checks, so the upgrade lengthens. Fewer than three nodes leaves no room to restart one node safely. | One-time design choice, re-checked each release |
| Dedicated hosts versus shared hosts | Keeper on dedicated hosts restarts without competing for memory or disk with ClickHouse servers, which shortens the window. Co-located nodes lengthen it. | One-time architecture choice |
| Keeper and server co-location in containers | Shared nodes or pods force coordinated restarts of both tiers, which adds steps and rollback risk. | One-time architecture choice |
| Size of replicated data | Larger replicated tables mean longer resync when a replica restarts, and each Keeper step must wait for that catch-up. | Recurs at every rolling step |
| Number of ClickHouse replicas and shards using the ensemble | More ReplicatedMergeTree tables and replicas depend on Keeper, so more components must be checked before each next step. | Grows over time, checked each release |
| Gap between current and target Keeper version | A larger gap means more changelog entries to read and more compatibility checks on snapshot and log formats. | Recurs at every release |
| Custom Keeper configuration | Non-default settings, such as tuned timeouts or ports, must be reviewed against the target version and may need changes before the upgrade. | One-time review, repeated when config changes |
| Depth of the backup and restore test | A tested restore of Keeper metadata and the replicated tables adds preparation time but shortens recovery if a step fails. Skipping it does not remove the risk. | Recurs before each upgrade |
| Monitoring between steps | Without clear health signals for quorum, leader status and replica sync, each step needs longer manual checks before the next node restarts. | One-time setup, recurring use |
How to verify the Keeper ensemble after the upgrade
Verification starts with the Keeper ensemble itself, then moves to the replicated tables that depend on it. Confirm that every Keeper node reports the target version, that the ensemble has exactly one leader, and that all expected members are present and following. A node that restarted on the new binary but has not rejoined the quorum is a failure even if the server process is up. Next, check the ClickHouse side: no replicated table should show readonly status, and replication queue and replica delay should be draining rather than growing. The ClickHouse replication guide describes how a recovered replica syncs missing data from a healthy replica, so a replica that stays behind after a restart is a signal to investigate, not to wait out.
Run a test insert into a replicated table after the restart, then confirm the row appears on every replica of that shard. Use a small, labeled dataset so the check is easy to clean up, and record the query, the timestamp and the counts returned from each replica. Restart each Keeper node once more, or confirm the restart sequence already performed, and verify that log and snapshot files are written to the configured directories and can be read on startup. A snapshot that is written but fails to load is only discovered at the next failure, so the check must include a clean restart of at least one node. Confirm the exact Keeper metrics, system tables and four-letter commands available in your version against the ClickHouse monitoring documentation for that release before you script these checks.
Keep a verification record for every Keeper change: the versions before and after, the node order, the output of each check, the test insert results and any warnings seen during the window. Store it with the change ticket so the next engineer can compare the baseline. Set alert thresholds on Keeper health before the next release, covering leader loss, lost quorum members, outstanding requests, and readonly replicas, so that the next upgrade starts from a monitored baseline rather than a manual spot check. Ask your DeployOpen contact or your own platform team to confirm which thresholds apply to your node count and workload.
Questions teams ask before upgrading Keeper
How much does a ClickHouse Keeper upgrade cost, and what is fixed versus recurring?
The cost depends on the written scope, so there is no standard figure to quote. A fixed-scope engagement covers the agreed work: reviewing the current Keeper and server versions, planning the order of restarts, running the upgrade in a pre-production environment, performing the production window, and checking the ensemble afterward. Those deliverables are agreed in writing before work starts. Recurring costs appear only if you buy an ongoing retainer for later upgrades, monitoring reviews or testing. Your own staff time, change board effort and any downtime planning are internal costs you carry either way. Ask any quote to separate one-time work from retainer items, and to state which environments are included.
How long does a Keeper upgrade take, and what drives the duration?
Duration depends mainly on the number of Keeper nodes, how many ClickHouse clusters depend on the ensemble, and how many environments must be tested first. Each node is restarted one at a time to keep quorum, so the window grows with node count and with the time each node needs to rejoin and sync. Other drivers include the size of the Keeper snapshot and log store, the number of replicated tables that must reconcile, the age of the current version relative to the target, and whether approvals or change windows limit when work can happen. Ask the provider to show the proposed restart sequence and the checks between steps, and confirm the timeline in writing.
Can our own team upgrade Keeper without a partner?
Yes, if your team has Linux operations experience, access to the Keeper hosts, and a non-production copy of the cluster to rehearse on. The ClickHouse documentation describes Keeper as the coordination service for ReplicatedMergeTree tables, and the upgrade steps are the same whether a partner or your staff run them. What usually decides the question is capacity and risk tolerance: whether engineers have time for rehearsal, whether someone can watch replication health during the window, and whether there is a documented rollback. A partner can supply the plan, the checks and a second pair of hands, while your team keeps ownership of the production change.
What does the work need from our staff, environments and change windows?
You need named owners for the Keeper hosts, the ClickHouse servers that depend on them, and the application teams that write to replicated tables. Engineers must have shell and sudo or equivalent access, and a pre-production environment with data and query patterns close to production is strongly recommended by the ClickHouse production guidance. Plan a change window long enough for sequential restarts and post-upgrade checks, with someone able to stop the process if quorum or replication degrades. Confirm freeze periods, backup schedules and on-call coverage before booking the window.
Do we have to upgrade Keeper in the same window as the ClickHouse servers?
No, Keeper can usually be upgraded separately, but the two versions must be compatible at each stage. Teams often upgrade the ensemble first, verify quorum and replication, and then roll the servers, or the reverse, depending on what the target ClickHouse release requires. Read the release notes for both components before choosing the order, and test the exact sequence in a pre-production environment. Splitting the work into two windows lowers the number of simultaneous changes, but it adds a period where mixed versions run. Document the chosen order and the checks that must pass before each step.
What does a rollback look like if the Keeper upgrade goes wrong?
A rollback means returning the affected Keeper nodes to their previous binary and configuration, then confirming the ensemble re-forms quorum and replicated tables resume syncing. Whether that is simple depends on whether the new version has written snapshot or log formats the older binary cannot read, so confirm downgrade support in the release notes before you start. Take a backup of Keeper data and of the ClickHouse tables before the window, since replication alone does not protect against corruption or human error according to the ClickHouse backup documentation. Agree rollback triggers, such as failed quorum after a restart, in advance so the decision is not made under pressure.
How do we check that the Keeper ensemble is healthy after the upgrade?
Confirm that every Keeper node reports as a member, that a leader is elected, and that the ensemble accepts reads and writes after each restart. On the ClickHouse side, check that replicas show no outstanding replication queue growth, that a recovered replica syncs missing data from a healthy peer, and that test inserts appear on all replicas. Compare logs for session errors and check disk space on Keeper hosts. Run a restore test from a recent backup on a spare cluster so you know recovery works, not just that the ensemble is up. Write these checks into the change record so the sign-off is repeatable.
How DeployOpen scopes and delivers a Keeper upgrade
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For a Keeper upgrade, we treat the ensemble as a separate change with its own quorum checks, rollback points and verification record.
How we scope
- We inventory every Keeper node, its host, its version and the ClickHouse servers that depend on it before quoting.
- The written scope names the target Keeper version, the change window, the ClickHouse servers involved and the evidence you will receive.
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before any change starts.
How we build
- We rehearse the rolling Keeper restart order on a pre-production ensemble with representative snapshot and log sizes.
- We check the target version's release notes and the ClickHouse production guidance against your current configuration and version list.
- We record quorum, leader election and session behavior at each step so every node change has a documented pass or stop decision.
How we hand over
- You receive the executed runbook, the rollback procedure and the exact commands and configuration changes used during the upgrade.
- The handover includes post-upgrade checks for ensemble health, replica readiness and ClickHouse replication status, with the results recorded.
- We walk your platform and on-call teams through the operating changes so they can run the next Keeper upgrade themselves.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the upgrade itself.
- A retainer can cover monitoring of Keeper health, review of new stable and LTS releases and planning of the next upgrade window.
- Ask us to confirm which supported version lines apply to your environment at the time of the scope, since these change over time.
