How to plan a rolling zero-downtime ClickHouse upgrade
A rolling upgrade of a replicated self-managed ClickHouse cluster is possible when the target version is on a supported path and the cluster has been rehearsed on representative data first. Zero downtime is a goal you plan toward, not a guarantee. Whether analytics keep running depends on three things: how many replicas each shard holds, whether client applications retry failed queries against another replica, and how far apart the old and new versions are. Teams upgrade for three reasons: to stay on a version that still receives security updates, to pick up bug fixes, and to avoid accumulating breaking changes that become harder to absorb the longer they wait. The ClickHouse security policy lists 26.9, 26.8, 26.7 and 26.3 as the versions receiving security updates, so any cluster running another line is outside that support window today.
Release cadence shapes the plan. According to the ClickHouse production FAQ, stable releases ship roughly monthly and LTS releases ship twice a year, with each LTS line supported for a year after release. The same FAQ recommends stable releases for most production use and reading the changelog before upgrading, because minor backward-incompatible changes can occur. Read the release notes for every version between your current build and the target, not only the target, and record each item that touches query syntax, settings defaults, table engines or the query analyzer. That list becomes the input for your test plan and for the decision on whether to pin any compatibility settings during the transition.
Before the first replica restarts, confirm three things in writing. First, that the target build has been run against a pre-production copy of the cluster with representative data and the query set your applications actually send. Second, that a verified backup exists, since replication protects against hardware loss but not against bad data or a failed upgrade. Third, that your client libraries and load balancers retry on connection loss and fail over between replicas, because that behavior, more than the server binary itself, determines what users see during each restart. Ask your platform team to document the expected query error rate during a test restart so the production go or no-go decision uses measured behavior rather than assumptions.
Which release changes affect a rolling upgrade
This table compares the release facts that shape a rolling upgrade, from the current stable and LTS lines to the breaking changes that can fail a replica mid-rollout. Read each row as a check to complete before you choose a target version and start the first replica.
| Change | Impact on a rolling upgrade | Action needed |
|---|---|---|
| Latest stable release v26.9.20.23-stable | Newest stable build with the mandatory analyzer. Mixed-version replicas during the rollout must still replicate with each other. | Confirm the target build on the ClickHouse releases page and read its changelog before scheduling. |
| LTS lines v26.8.29.10-lts and v26.3.53.4-lts | Both were published October 16, 2026. An LTS target keeps the cluster on a line that receives support for a year after release. | Decide whether the target is an LTS line or the latest stable, and record that choice in the change plan. |
| Monthly stable releases with three latest receiving backports | Only the three most recent stable releases get bug fix backports. Waiting too long can leave you on a line that no longer gets fixes. | Schedule the upgrade within the backport window and confirm the current list of supported versions in the security policy. |
| LTS supported for one year | The support window limits how long a cluster can stay on one LTS line and sets the pace of required upgrades. | Track the end-of-support date for your current line and plan the next rolling upgrade before it arrives. |
| Minor backward-incompatible changes | Even within a line, a minor release can change behavior that breaks queries, settings or replication between replicas. | Read the changelog for every release between your current and target versions and test on representative data first. |
| Mandatory analyzer in 26.9 | Old query analysis is removed, so queries that relied on it can fail or return different results on upgraded replicas. | Run representative queries on a 26.x test cluster, fix HAVING, JOIN and alias incompatibilities, and review migration aids before upgrading. |
| Mixed-version replication during rollout | Replicas on different versions coexist for hours. Expected checksum warnings may appear in logs while the rollout proceeds. | Confirm with the vendor which mixed-version warnings are expected for your path, and set alert thresholds that do not page on them. |
Why replicas can run mixed versions during the upgrade
A rolling upgrade works because replicas of a ReplicatedMergeTree table coordinate through ClickHouse Keeper, not through the server version they run. Keeper, which replaces ZooKeeper, holds the replication metadata that tells each replica which data parts exist and which must still be fetched. When one replica is stopped, upgraded and restarted, it rejoins the group and, according to the ClickHouse replication guide, automatically syncs missing data from a healthy replica without manual intervention. This is the mechanism that lets the rest of the cluster keep serving queries while one node is down, and it is why the upgrade is planned one replica at a time.
The consequence is that mixed versions exist for the length of the window: some replicas run the old release while others already run the new one. During that period, replicas may write or merge parts in ways the other version reads with differing expectations, so checksum or part-consistency warnings can appear in logs even when the cluster is healthy. Treat such warnings as expected only after you have confirmed they are tied to the version difference and do not indicate data loss, and keep a record of each occurrence with the replica name and both versions involved.
Do not assume which mixed-version behavior applies to your pair of versions. Read the release notes and changelog for both the source and target releases, and ask ClickHouse, through its documentation or support channels, which replication and checksum behaviors are supported between them. Confirm the result in a pre-production cluster with representative data before the production window opens, and set the stop condition in advance: if a warning appears that the release notes do not describe, halt the rollout and resolve it before upgrading the next replica.
What to prepare before the first replica restarts
Every replica restart carries risk, so the work starts before any binary changes. The six items below establish a recovery point, prove that recovery works, reveal what the new version will break, and fix the conditions under which you would stop. Complete them in order, and record the outcome of each in the change ticket.
Take a verified backup
Run BACKUP to local disk, S3-compatible storage or Azure Blob Storage before the upgrade starts, so you hold a copy that does not depend on the cluster you are changing. Replication does not protect against human error or corruption, because a bad write or dropped table replicates to every copy. Record the backup name, its location and the time it completed.
Prove the backup restores
Restore the backup to a spare cluster and run row count and checksum comparisons against the source tables. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when it is needed. Time the restore so you know how long recovery would take if the upgrade fails.
Inventory queries and settings
List the queries, user-level settings, profiles and custom configuration files your applications depend on, including dashboards and scheduled jobs. Flag anything that relies on behavior the release notes mark as changed, such as aliases, HAVING clauses or JOIN handling. This list becomes the test set for the pre-production run.
Test on representative data
Run the upgrade in a pre-production environment loaded with data that matches production volume, schema and query mix. The ClickHouse production FAQ recommends this step and reading the changelog before upgrading, because minor backward-incompatible changes can occur. Compare query results and error logs against the current version.
Confirm Keeper health
Check that the ClickHouse Keeper ensemble has at least three nodes, that a leader is elected and that every member reports healthy before you restart any replica. The replication guide recommends dedicated hosts in production, with about 4 GB of RAM per node. Losing quorum during a rolling restart stops writes to replicated tables, so confirm the spare capacity you need.
Write the rollback decision point
Define in writing the conditions that stop the rollout, such as failed queries on the test set, replicas that do not resync, rising errors on the first upgraded replica, or a Keeper warning. Name the person who makes the call and the time limit for the decision. Confirm with ClickHouse release notes whether a downgrade is possible once new features are used.
The order of shards and replicas during the upgrade
This sequence moves a replicated ClickHouse cluster to a new version one replica at a time, so that every shard keeps a working copy of its data and every step has a verified state to return to.
Confirm the target version against support lists
The platform owner checks the target build against the ClickHouse security policy and the current release notes, then records the chosen version, whether it is a stable or LTS line, and the release notes reviewed for backward-incompatible changes. The security policy lists which versions receive security updates, so a target outside that list is a blocker, not a footnote. Done means a signed change record names one version and links the changelog entries that were read. Confirm the exact upgrade order with the target release notes before any host is touched.
Take and verify backups before any change
The database team runs a full BACKUP of the affected databases and tables to storage outside the cluster, then restores a sample into a spare environment to prove the archive is usable. Replication keeps replicas in step but does not protect against deletion or corruption, so this backup is the rollback anchor for the whole run. Done means the backup job reports success, the restore test returns the expected row counts, and the backup location and identifier are written into the change record. Record the configuration files and Keeper state as well.
Check cluster health and replication queues first
Before the first restart, the operator confirms that every replica is reachable, that Keeper has quorum, and that replication queues are empty or stable. An upgrade started on a cluster that is already lagging makes it hard to tell upgrade effects from existing faults. Done means every replica reports healthy, the replication queue depth is recorded as a baseline, and no replica is in read-only mode. Pause the run if any of these checks fail.
Upgrade one non-leader replica in a single shard
The operator picks one replica in one shard that is not the current leader, stops it, installs the target version using the team's packaging method, and starts it again. Starting with a non-leader keeps write coordination on a version the cluster already knows. Done means the node reports the target version, rejoins the cluster, and appears in the replica list. Exact restart commands depend on how ClickHouse was installed (Linux packages, containers or Kubernetes operator), so the runbook must name them for that method.
Wait for queues to drain and checks to pass
After the upgraded replica rejoins, the operator watches its replication queue until it drains and compares row counts or checksums for the replicated tables against a healthy peer. A recovered replica syncs missing data from a healthy one automatically, so the work here is observation. Some checksum warnings can appear while versions differ, and the team must check them against the target release notes before treating them as corruption. Done means the queue is empty, query and insert tests pass on the upgraded node, and no new errors appear in the server logs.
Repeat across replicas and then shards
The team repeats the same cycle for each remaining replica, one at a time, keeping at least one healthy replica for every shard online at all times. Leaders are upgraded last within each shard, after a failover moves leadership to an already-upgraded replica. Done means each replica passes the same checks and the change record logs the order, the time of each restart and the queue state after it. Pause and fall back to the backup point if a replica fails to rejoin or the queue stops draining.
Verify Distributed tables and client connections
Once every shard runs the target version, the operator runs test queries through each Distributed table and through the application's own connection path, including inserts, aggregations and the queries that depend on changed behavior. Done means results match the pre-upgrade baseline, query errors are absent from the logs, and client drivers connect without warnings. Record the final version on every node so the cluster state is confirmed in one place.
Run post-upgrade checks and close the change
The operator reviews monitoring for elevated errors, slow queries and replication lag over an observation period agreed in the change plan, then confirms backups still run on the new version. Done means the observation window ends with no open alerts, the backup job completes on the new version, and the change record is signed off by the platform owner. Keep the pre-upgrade backup until the team has agreed when it can be retired.
How the path changes for Kubernetes, packages and restricted networks
On bare hosts or virtual machines installed from Linux packages, the upgrade is a sequence of package updates and service restarts that the team runs itself. Each replica is stopped, its package is upgraded, the service is started, and the team waits for the replica to catch up before moving on. Nothing here is automated by the platform, so the discipline lives in the runbook and in who holds the change window.
- Package pinning: Confirm the exact package version on every host before the first step, so mixed fleets are visible rather than assumed.
- Service control: Use the same init or systemd unit names on every node so the restart command is identical across the shard.
- Config drift: Compare the server configuration files between replicas before and after each step, because a single outdated file can change behavior mid-rollout.
- Replication lag checks: Read replication queue and readonly flags from the system tables after each restart before selecting the next replica.
- Rollback packages: Keep the previous package files in a local repository so a rollback does not depend on the public mirror being reachable.
Before starting, confirm which package repository your hosts use, whether it is the vendor's public repository or an internal mirror, and who approves each host restart.
In Kubernetes, the rolling method is delivered by the cluster controller, which restarts pods in a defined order and waits for readiness. The image tag decides the version, so the change control moves from package management to manifest or Helm values review. Readiness probes must reflect real replication health, or the controller will advance before a replica has caught up.
- Image sourcing: Pull the server image from a registry you control and pin it by digest, not by a floating tag.
- Update strategy: Confirm whether your operator performs ordered replica restarts or whether you must partition the StatefulSet update yourself.
- Readiness signals: Check that the probe covers query availability and replication status, not only that the port answers.
- Keeper pods: Upgrade Keeper on its own schedule, since its pods carry quorum and must not restart together with server pods.
- Persistent volumes: Verify that data volumes survive pod rescheduling and that the storage class supports the restart pattern you plan.
Confirm your operator version and its documented upgrade behavior for ClickHouse clusters, since the controller's ordering rules decide how much of the sequence you must script yourself.
In networks without outbound internet access, the release artifacts must be mirrored into an internal repository or registry before the upgrade starts. The mirror becomes part of the supply chain, so the team needs to know exactly which build it is installing and where that build came from. Downloads, image pulls and documentation lookups must all work from inside the boundary, and the changelog must be reviewed from a copy taken before the change window.
- Artifact mirroring: Copy the chosen release packages or images into the internal repository and record checksums at import time.
- Version selection: Choose a version that appears on the supported list in the security policy at the time of the change, and confirm the release date on the releases page.
- Offline documentation: Save the changelog and upgrade notes locally, because the public pages may not be reachable during the change.
- Verification tooling: Confirm that monitoring, query clients and test harnesses also run from inside the network and do not fetch dependencies online.
- Change approval: Ask your security team how mirrored artifacts are scanned and signed before they enter production.
Confirm with your security and platform owners how mirrored artifacts are approved, and whether the restricted network has its own staging cluster where the same steps can be rehearsed first.
What to verify after each replica and after the full cluster
Each replica restart is only safe once the cluster shows it is healthy again, and the full cluster is only done once queries, clients and recovery paths all behave as before. The cards below set the checks to run at both points, in the order an operator would reach them.
Replication queue and replica lag
After a replica comes back, confirm it has caught up before touching the next one. A recovered replica syncs missing data from a healthy peer on its own, so the work is watching that the queue drains and lag returns to zero rather than assuming it has. Check the replication queue and absent or stuck parts on every replicated table, and hold the rollout if the queue keeps growing. Record the timings so later replicas can be compared against the first.
Keeper quorum after each restart
ClickHouse Keeper coordinates every ReplicatedMergeTree table, so a lost quorum stalls writes across the cluster. Confirm that the Keeper ensemble reports a leader and all expected members are in sync before and after each replica restart. The guide recommends at least three Keeper nodes on dedicated hosts in production, and a rolling restart should never take a second Keeper node down while another is out. If you upgrade Keeper separately, verify the ensemble again once it is done.
Query results against the pre-upgrade baseline
Before the first restart, capture results from a fixed set of production queries, including aggregations, joins and queries over recent partitions. Rerun the same set after each replica and after the full cluster, and compare row counts and values, not just that the query succeeds. Differences usually point to analyzer behavior, changed defaults or a replica that has not finished syncing, so treat any mismatch as a stop condition until it is explained.
Error logs and checksum warnings
During the mixed-version window, some replicas may log checksum mismatch warnings while parts are merged or fetched across versions. Review server error logs on every replica for warnings and errors that appear after a restart, and separate expected mixed-version messages from new failures. Confirm that replicated parts converge, and escalate any error that repeats after the cluster is uniform, since expected warnings should disappear once every replica runs the new version.
Client and application compatibility
Applications, BI tools and ingestion jobs can fail on a new release even when the server is healthy. Run your application test suite and representative dashboards against the upgraded cluster, paying attention to queries that rely on HAVING, JOIN behavior and column aliases, which the mandatory analyzer affects. Check that drivers and connection settings still negotiate correctly, and confirm that scheduled jobs complete within their usual window.
A backup taken after the upgrade
Once every replica runs the new version and the checks above pass, take a fresh backup before returning the cluster to normal change control. Replication does not protect against human error or corruption, and a backup gives you a known-good point to restore from if a later problem appears. Store it outside the cluster, and schedule a test restore on a spare cluster, because a backup that has never been restored may not restore properly when needed.
What makes a ClickHouse rolling upgrade longer
This table lists the factors that stretch a rolling upgrade of a replicated ClickHouse cluster and the effect each has on effort. Use it to see which details your quote must reflect before work starts, since most of these drivers are only visible once the cluster has been inventoried.
| Driver | Effect on effort | What to ask for in a quote |
|---|---|---|
| Number of shards and replicas | More nodes mean more restart cycles, more health checks between steps and a longer rollback window if a node misbehaves. | Node count per shard, replica placement, and the restart order the partner proposes. |
| Data volume and replication lag | Large parts and slow replicas take longer to resync after restart, so each step waits on catch-up before the next node can go down. | Current data volume per replica, observed replication queue behavior, and how catch-up will be checked. |
| Number of version steps | Each step between the current and target release adds changelog review, test cycles and possibly intermediate upgrades. | The exact source and target versions, and whether intermediate releases are included in the plan. |
| Custom queries affected by breaking changes | Queries that depend on removed or changed behavior, such as those affected by the mandatory analyzer, need rewriting and retesting before the upgrade. | A query inventory method, the number of queries reviewed, and who signs off on rewritten queries. |
| Client applications needing retesting | Each dependent application, driver or BI tool needs its own regression test against the target version. | A list of connected clients, the test environment used, and the acceptance criteria per application. |
| Keeper topology | If Keeper runs with the servers or is upgraded separately, the plan needs extra quorum checks and sequencing around each restart. | Keeper node count, hosting (dedicated or shared), and whether Keeper upgrades are in scope. |
| Backup size and restore test duration | Taking a pre-upgrade backup and proving it restores on a spare cluster adds time that scales with data size. | Backup target and size, the restore test environment, and the evidence delivered after the test. |
| Deployment method | Upgrades on Kubernetes, containers or Linux packages follow different procedures, and restricted networks add artifact transfer steps. | The deployment method in use, how packages or images reach the cluster, and the change tooling required. |
Questions teams ask before upgrading ClickHouse in place
Can a ClickHouse cluster be upgraded in place without downtime?
In most replicated setups, yes, because each replica can be restarted on the new version while the others keep serving queries, but whether it is truly without interruption depends on how the cluster is built. The determining factors are whether every shard has at least two replicas, whether the Distributed tables and applications can fail over between them, whether ClickHouse Keeper keeps quorum during restarts, and whether the application's query patterns still work on the new release. Single-replica shards, a single Keeper node, or clients that connect to one fixed host will cause interruptions. Confirm each of these in your own environment before you schedule the work, and rehearse the sequence on a pre-production cluster with representative data.
What does a ClickHouse upgrade cost, and which parts are one-time?
The cost is mostly labor, split across planning, pre-production testing, the production rollout, post-upgrade verification and documentation. Those are one-time costs for a given upgrade. Recurring costs appear only if you keep a retainer for ongoing patching, upgrade cycles and monitoring. Other cost drivers include the size of the query inventory, how many clusters and environments exist, whether Keeper is upgraded in the same window, and how much test infrastructure you need to stand up. Ask any provider, including DeployOpen, for a quote built from a written scope that lists these items separately, along with what the client must supply, such as spare capacity and access. DeployOpen does not publish a price list.
How long does a rolling ClickHouse upgrade take, and what drives the duration?
Duration depends on the number of shards and replicas, the data volume each replica must catch up on after restart, the size of the query inventory to re-test, and how many environments must be upgraded first. The release notes and changelog review, the pre-production run and the post-upgrade checks usually take longer than the restarts themselves. Replicas that were offline sync missing data from a healthy replica automatically, so catch-up time grows with the amount of data written during the restart window. Ask for a timeline that breaks out each phase, and confirm change windows with your operations team before agreeing on a date.
What must our team provide before an upgrade can start?
A rolling upgrade needs several inputs from your side before any work begins:
- Administrative access to the cluster hosts, Keeper nodes, configuration management and the monitoring stack.
- A pre-production cluster with representative data volumes and recent schema changes.
- An inventory of the queries, dashboards, ingestion jobs and client libraries that touch ClickHouse, with owners named for each.
- Agreed change windows, rollback approvers and a contact for application teams.
- A current, tested backup, since replication alone does not protect against human error or corruption.
Missing any of these usually adds time to the project, so confirm them in the written scope.
Can our own team upgrade ClickHouse, and what do the release notes require?
A capable platform team can upgrade ClickHouse itself, provided it reads the changelog for each version in the path and tests against representative data before touching production. The ClickHouse production guidance recommends stable releases for most production use, a pre-production environment with representative data and queries, and reading the changelog before upgrading, because minor backward-incompatible changes can occur. Check the security policy to confirm your target version is still supported. Decide early who owns the query regression testing, since that is usually the largest part of the work. A partner is most useful when the team lacks spare capacity, has several clusters to coordinate, or needs the work documented as evidence for an assessment.
Should we upgrade from 26.3 directly or through intermediate versions?
Check the supported list first. The ClickHouse security policy lists versions 26.9, 26.8, 26.7 and 26.3 as currently supported with security updates, and all other versions as unsupported, so an upgrade that starts from 26.3 has a clear supported starting point. Whether to step through an intermediate release depends on the changelogs between your current version and the target, the breaking changes they list, and how far your query inventory has drifted. Moving between LTS lines involves its own planning, and your team should review those notes against your queries before choosing a path. Confirm the path in a pre-production run, not on paper alone.
What happens if we need to roll back after an upgrade?
Rollback is possible only under conditions you should check before the upgrade starts. A replica can often return to an earlier binary if the new version has not written data or metadata that the older one cannot read, and if no new features have been used that block downgrade. Before any rollback, take a BACKUP of the affected tables, since the backup documentation says a backup never test-restored will probably not restore properly when needed. Keep the rollback plan in writing with named approvers, the decision criteria and the order in which replicas return. Test that path on the pre-production cluster as well, because a rollback you have not rehearsed is unlikely to be fast or clean.
How DeployOpen scopes and delivers a ClickHouse upgrade
DeployOpen plans and runs rolling ClickHouse upgrades inside infrastructure you control, starting from a written scope and a pre-production copy of your cluster. We work through the replica order, version checks, verification and handover so your team can run later upgrades with the same runbook. Engagements are fixed-scope, and ongoing support is available as a retainer.
How we scope
- Inventory your current version, cluster topology, Keeper ensemble and the applications that query it.
- Confirm the target release against the supported version list and the changelog for backward-incompatible changes.
- Agree in writing which replicas, shards, Keeper nodes and queries are in scope, plus the acceptance criteria.
How we build
- Rehearse the full rolling sequence on a pre-production cluster with representative data and queries.
- Write the replica-by-replica runbook, including checksum warnings expected during mixed versions and stop conditions.
- Test a backup and restore on a spare cluster before the first production replica is touched.
How we hand over
- Deliver the tested runbook, rollback decision points and post-upgrade check scripts as documented artifacts.
- Walk your operators through each step, including replica health, replication queue and query result comparisons.
- Record the final versions, configuration changes and open items so the next upgrade starts from a known state.
How we support
- Offer a retainer for ongoing upgrade support, patch planning and reviews of new stable or LTS releases.
- Repeat the rehearsal and runbook update for each later upgrade within the agreed support scope.
- Confirm with your team which support channels, response expectations and review cadence apply before work starts.
