Rolling back a ClickHouse upgrade: when it is possible
Rollback is not something to assume when a self-managed ClickHouse cluster moves to a newer release. Whether you can return to the old version depends on what the new release wrote to disk and whether your team turned on features that only the new version understands. Decide this before the upgrade window opens, not after a failed rollout has already changed your data. Start with the target you would roll back to. The security policy lists 26.9, 26.8, 26.7 and 26.3 as the versions currently receiving security updates, and all other versions are unsupported. Stable releases ship roughly monthly, and LTS releases ship twice a year and are supported for a year after release, so your rollback target should sit on one of those supported lines. Confirm the exact downgrade behavior of each target version in its changelog and with the vendor, because that behavior is not something this section can state for you.
The practical question is which version your cluster would return to and what it would have to read. Write down the current version, the target version and the rollback version for every node, and check whether the target release changed the on-disk format of data parts or Keeper metadata. Keep in mind that a mixed-version cluster during a rolling upgrade is itself a state you need to plan for, and that a rollback may need to pass through the same mixed state in reverse. Finally, record whether the new release was run with any new features or changed defaults enabled, including settings that switch on new behavior. Reading the release notes and the production FAQ before the upgrade tells you what to check, and confirming each item with the vendor gives you the answer your rollback plan depends on.
Which ClickHouse changes make a rollback harder
This table compares the kinds of ClickHouse change you will meet during an upgrade and how each one affects your ability to return to the previous version. Read each row against the target release, because the rollback cost depends on whether the change touches data formats, query behavior or access configuration.
| Change | Impact on rollback | Action needed |
|---|---|---|
| Stable and LTS release lines (26.9 stable, 26.8 LTS, 26.3 LTS) | Moving between lines changes which build you can fall back to. Versions outside the supported list receive no security updates, so a rollback target may be unsupported. | Check the releases page and security policy for the current supported list, then pick a target version you can return to safely. |
| Minor backward-incompatible changes | The production FAQ warns these can occur in minor releases, so a binary rollback may not restore prior query or setting behavior. | Read the changelog for every version between your current and target release, and test on representative data first. |
| Mandatory analyzer in 26.9 | Affects queries, not just binaries. Queries that depended on the old analysis can fail or return different results on 26.9, and reverting the binary does not fix application code changes. | Test HAVING, JOIN and alias usage on 26.x before upgrading, and confirm any compatibility aids in the 26.9 changelog. |
| New table features or settings that write new data formats | Once a feature writes new data parts or metadata, an older binary may be unable to read those tables, which blocks a simple downgrade. | Identify features enabled in the target release notes, keep them off until rollback is no longer needed, and confirm reader compatibility in the changelog. |
| Access control and configuration file changes | Changed defaults or config keys can alter authentication, users, roles or profiles. Reverting the binary may not revert edits made to SQL-driven access objects during the upgrade. | Export users, roles, row policies and config files before upgrading, and compare them against the target release documentation. |
| Replication and Keeper metadata | Upgraded replicas and Keeper state may not match an older binary after writes resume, so rollback can require restoring from backup rather than reverting one node. | Confirm the Keeper and replica procedure in the replication guide and plan the rollback path before the first replica is upgraded. |
Why using new features blocks a downgrade
Assume a downgrade is possible only until something written by the new version depends on that version. The old binary may not be able to read data, metadata or settings that the new one wrote, so a rollback that looks simple on a test cluster can fail on production tables. The risk does not come from the binary alone. It comes from what the upgraded cluster has stored since the upgrade started: table definitions in ZooKeeper or Keeper, parts on disk, and persisted server or table settings.
The most common blockers are features that change what is written to disk or to metadata. A new table option, a new data type, or a new storage or engine setting can cause the old version to refuse to start, or to skip a table it cannot parse. Changed defaults are a quieter risk: when a new default is persisted into a table definition during CREATE TABLE or ALTER TABLE, the old server reads a setting it does not understand. DDL run on the new version can also write metadata in a format the earlier release does not accept. Before rolling back, list which of these features your teams have used since the upgrade, and check the target release notes for format or metadata changes.
Do not treat a downgrade path as supported until the vendor confirms it for your exact versions. Read the changelog entries for every release between your current and previous version, and confirm the downgrade question with ClickHouse, using your own table definitions and a sample of the features you have enabled. If the answer is unclear, plan the rollback around a restore from backup onto the older version, and test that restore on a spare cluster before the production upgrade.
Preparation before upgrading ClickHouse: backups and restore tests
Before any ClickHouse upgrade, a rollback is only as good as the backups and restore tests behind it. The cards below cover the artifacts to produce, from a full backup and a tested restore to the configuration, access inventory, pre-production run and named owner that let you decide quickly if something goes wrong.
Full backup with BACKUP
Take a full backup of every table you cannot rebuild using BACKUP to local disk, S3-compatible storage or Azure Blob Storage, before the first node changes version. Incremental backups can reduce the work for later snapshots, but the pre-upgrade copy should be a complete, independently stored set. Replication does not protect against human error or corruption, because a bad DROP or a corrupted part is copied to every replica. Evidence produced: the backup ID, its storage location, and the list of databases and tables it covers.
Restore test on a spare cluster
The backup documentation warns that a backup never test-restored will probably not restore properly when needed. Restore the pre-upgrade backup onto a spare cluster built from the same version and configuration, then run row count and checksum comparisons against the source. Record how long the restore took and which steps needed manual work, because that is the real recovery time you will plan around. Evidence produced: a signed restore report with table-level comparisons.
Access control inventory
Export users, roles, row policies, settings profiles and quotas, along with external authenticator settings for LDAP, Kerberos, SSL X.509, HTTP and JWT where they are used. Access objects are stored in SQL and can be dropped or altered by mistake, and a rollback that restores data but not grants leaves applications failing. Confirm the default user is restricted before the change. Evidence produced: a dated export of access objects and a checksum of the file.
Server and Keeper configuration copy
Copy the server configuration, cluster definitions, macros, storage policies and TLS material, and the Keeper configuration and its ensemble membership. Store the copies under version control or in a dated archive that the rollback team can reach without the failed hosts. Include the exact package or container image tags in use, so the previous binary can be reinstalled. Evidence produced: a configuration snapshot tied to a version tag and a host inventory.
Pre-production run with representative data
The production FAQ recommends a pre-production environment with representative data and queries, and reading the changelog before upgrading, since minor backward-incompatible changes can occur. Replay your most important dashboards, scheduled queries and ingestion jobs against the target version and compare results with the current release. Any query that changes output or fails becomes a go or no-go item. Evidence produced: a test log listing each query, its result comparison and the decision taken.
Rollback owner and change window
Name one person who can declare a rollback and a second who can execute it, and agree the change window, the rollback trigger conditions and who communicates with application owners. Write down the exact steps and the time after which rollback becomes more expensive than fixing forward. Rehearse the steps on the spare cluster so the window is realistic. Evidence produced: a signed rollback runbook with named roles, triggers and the approved change window.
Rollback options by situation
A single replica that fails after an upgrade is usually the easiest case, because the rest of the shard still holds a healthy copy of the data. The decision is whether to repair the node in place or rebuild it. ReplicatedMergeTree coordinates replicas through ClickHouse Keeper, and according to the replication guide, a recovered replica automatically syncs missing data from a healthy replica without manual intervention. That makes rejoining the normal first step, provided the failure is confined to one node and the other replicas are serving correct data.
- Preconditions: At least one other replica of the same shard is healthy, Keeper has quorum, and the failed node's disks are intact or can be replaced.
- Rejoin first: Restart the node on the same ClickHouse version it was running before the failure, check the replication queue, and confirm it catches up.
- Replace when disks are suspect: If the disks or local data are damaged, clear the local replica state and let it resync from a healthy replica instead of trusting the old files.
- Do not roll back the whole cluster: Downgrading a single node to an older binary is generally not supported once data has been written in the new format, so confirm this with the vendor first.
- Decision owner: The platform or database on-call engineer, with the cluster owner informed.
Before acting, confirm with ClickHouse support or the release notes whether the failing version has a known defect, and check that the healthy replicas are not showing the same error.
A regression that appears during a rolling upgrade is different because the problem may already exist on the replicas you have upgraded. Each shard or the whole cluster can be running a mix of versions, and the mixed state itself can produce warnings and behavior that does not occur on a uniform cluster. Two responses are available: stop the rollout and hold the mixed state while you investigate, or fix forward by applying a patched release. Downgrading every upgraded node is the least predictable option and should be treated as a last resort.
- Preconditions: You know which shards and replicas run the new version, and you have a verified backup taken before the rollout started.
- Stop and hold: Pause the rollout, leave the cluster in a known mixed state, and stop any further schema or setting changes.
- Fix forward: Move to a later patch release from the same line if the vendor confirms a fix, and test it on a staging cluster first.
- Check new features in use: If the upgraded version has already been used to write new formats or enable new features, a downgrade may not be possible, so confirm this before deciding.
- Decision owner: The platform lead, with input from the application owners who see the regression.
Before choosing between hold and fix forward, ask the vendor whether the regression is known, whether a patch exists for the target line, and whether a downgrade of the upgraded nodes is supported for your exact versions.
Data-level problems are the most serious, because replication copies whatever is written to every replica. If a bad query, a faulty migration or a corrupted part reached all replicas, the healthy copies are not a safe source. The backup documentation states that replication alone does not protect against human error or corruption, so recovery means restoring from a verified backup into a clean cluster and then replaying or reconciling data written after that backup. This path is usually owned by the data owner and the security or incident lead, not by the rollout engineer alone.
- Preconditions: A backup exists that predates the damage, it has been restored successfully in a test environment, and you know its exact contents and timestamp.
- Restore into a clean cluster: Use BACKUP and RESTORE to rebuild the affected tables on a new cluster rather than overwriting the damaged one.
- Identify the window: Establish when the damage started and which writes after the backup must be replayed from the source systems.
- Preserve evidence: Keep the damaged cluster intact until the incident owner agrees it can be decommissioned, since it may be needed for root-cause analysis.
- Decision owner: The incident lead and data owner, with legal or compliance involved if personal data is affected.
Before restoring, confirm with ClickHouse support the restore steps for your version, how the backup handles the affected table engines, and whether any part of the damage can be repaired without a full restore.
Rolling back a ClickHouse upgrade in production, step by step
This sequence turns a ClickHouse upgrade into a controlled change with a verified backup, a canary replica, written checks and a stop condition, so the team knows before each expansion whether rollback is still possible.
Confirm current and target versions against support
Before anything else, record the exact running version on every server and the version you plan to install. Check both against the ClickHouse security policy, which lists the versions that receive security updates, and against the GitHub releases page. Read the changelog between the two versions for backward-incompatible changes, since the production FAQ warns that minor releases can contain them. Output: a signed-off change record listing current version, target version, support status and the changelog items that touch your queries. If the current version is unsupported, treat that as a reason to act, but plan the path with the upgrade guidance for versions more than a year old.
Take and verify a restorable backup
Create a full backup of the affected databases and tables with BACKUP to storage that is separate from the cluster, such as S3-compatible or Azure Blob Storage. Record the backup name, its size and the list of tables it covers. Then restore it to a spare cluster or a scratch database and compare row counts and checksums against production for a sample of large and recently written tables. The backup documentation warns that a backup never test-restored will probably not restore when needed, so the output is a restore test that passed, not just a backup job that finished. Replication alone does not cover deletion or corruption, so do not treat the replicas as the backup.
Freeze the rollout and record cluster state
Pause all other schema changes, large merges you control, and deployment of application releases that depend on new query behavior. Capture the cluster state while it is still healthy: the system.replicas view for replication lag and readonly flags, Keeper health, the settings in use and the server configuration files. Store this snapshot with the change record. Record the state before reverting too, because a later rollback decision needs to know what was healthy when the upgrade began. Output: a frozen change window and a dated state snapshot that another engineer could read without asking you.
Upgrade one replica as a canary
Choose one replica in a shard that has healthy peers, upgrade its ClickHouse server package or container, and restart it. Confirm it rejoins, catches up missing data from a healthy replica and reports no readonly tables. Keep the other replicas on the old version while the canary runs. Expect mixed versions during this window and follow the sibling guidance on rolling upgrades for the order of shards and replicas. Output: a canary that has served traffic for a period you agree on in the change record, with no new errors in the server log.
Run representative queries and compare results
Replay a fixed set of production-representative queries against the canary and against an old-version replica, using the same data. Include aggregations, joins, HAVING clauses and queries that use aliases, since these are the areas most affected by analyzer changes. Compare result sets, row counts, error messages and latency. Investigate any difference before continuing. Output: a comparison report signed by the application owner that lists each query, whether results match and any accepted differences with a reason. If you pinned any compatibility settings, record them here.
Expand the rollout only after checks pass
Upgrade the remaining replicas one shard at a time, repeating the restart and catch-up checks from the canary step on each node. After each node, confirm replication lag is back to normal, no tables are readonly and query error rates match the baseline. Do not start the next node until the previous one passes. Output: all replicas on the target version, a post-upgrade health check recorded against the snapshot from the freeze step, and the change window closed in the change record.
Handle Keeper or replica problems during the rollout
If a replica falls behind, stops accepting writes or logs replication errors, stop the rollout immediately and leave the remaining nodes on their current version. Check Keeper quorum and the health of the other Keeper nodes before touching ClickHouse servers again. If Keeper itself is unhealthy, do not improvise a Keeper upgrade mid-rollout, since Keeper upgrades are handled separately and need their own plan. Output: a recorded incident note with the time, the symptom, the state snapshot and the decision to pause, continue or revert.
Apply the written stop condition and revert if needed
Agree the stop condition before the rollout begins: the specific errors, lag thresholds or result mismatches that halt it, and who has authority to call a rollback. If the condition is met, return affected nodes to the previous version only if the change record confirms that the new version has not written data or features the old version cannot read, which is why feature use blocks downgrades. If that cannot be confirmed, restore from the verified backup to a clean environment instead. Output: a rollback decision record and a post-rollback check that replicas are healthy and results match the pre-upgrade snapshot.
What drives rollback effort and duration
This table compares the factors that make a ClickHouse rollback longer or harder, and what each one changes in the work. Read each row as a question to answer about your own cluster before you agree a rollback window. The main takeaway is that data volume, the spare-cluster setup and the state of your runbook usually matter more than the version numbers involved.
| Driver | Why it changes the work | What to ask for |
|---|---|---|
| Data volume and restore time | A full backup and restore test must copy and reattach every part in scope, so restore time grows with data size and storage throughput. A rollback that depends on restoring from backup is only as fast as the last restore you measured. | A measured restore time from a recent test on comparable hardware, and the backup type (full or incremental) that the plan depends on. |
| Number of shards and replicas | Each shard and replica must be checked for version state, replica sync and Keeper health. Mixed versions across many nodes multiply the checks and the chance that one node is left behind. | A node-by-node rollback order, the health checks at each step, and the criteria for stopping a rollback partway through. |
| Customizations (UDFs, dictionaries, settings profiles, query patterns) | User-defined functions, external dictionaries, custom settings profiles and application query patterns each need to be confirmed on the target version. Each one adds a compatibility question to the rollback test. | An inventory of custom objects and application queries, with the version each one was last tested against. |
| Analyzer-related query changes | If queries relied on the old analyzer and were changed to work on the new one, a rollback can make those queries fail or return different results. Rolling back the binary does not undo changes made to schemas or application SQL. | A list of queries or views edited for the new analyzer, and the owner who will decide whether they stay on the new version. |
| Use of new features and new data formats | Once a cluster writes data or metadata in a format or uses a feature added in the newer version, older binaries may not read it. This is the most common reason a rollback stops being a simple binary change. | A list of features and table settings enabled after the upgrade, and the vendor or release note that confirms whether an older binary can read them. |
| Spare cluster for restore tests | Restore tests on production hardware compete with live traffic and carry risk. A spare cluster lets the team prove the backup restores and the rollback steps work before they are needed. | Whether a spare cluster exists, its size relative to production, and how quickly it can be refreshed with a production backup. |
| External authenticators (LDAP, Kerberos, JWT, certificates) | Authentication paths behave differently after a version change and during a rollback. Each authenticator that is not covered by tests adds time to verify that users can still sign in after the rollback. | A test record for each authenticator on the target version, and a documented fallback user or certificate path for emergencies. |
| Monitoring and runbook maturity | Clear dashboards, alerts and a written runbook shorten the time to decide and act. Without them, the team spends the rollback window working out what state each node is in. | A written rollback runbook with named decision owners, a rehearsal record, and the metrics used to confirm each step succeeded. |
Questions teams ask before rolling back a ClickHouse upgrade
Can we always downgrade ClickHouse after an upgrade?
No. A downgrade is possible only when the target version can read the data and metadata the newer version has written, and when no new feature has changed on-disk formats or table definitions in a way the older binary cannot handle. The main blockers are new table engines or settings used in production tables, changed part formats, schema changes made after the upgrade, and replicas that have already been upgraded and written data in a newer layout. Whether a given path is safe depends on the exact source and target versions, so confirm each pair against the changelog and your own test cluster before you plan around it. Treat a rollback as a planned recovery path, not a guaranteed undo, and make your backup plan the fallback when a downgrade is not supported.
What does a ClickHouse rollback cost, and what drives it?
The cost is mostly labor and spare capacity, not licenses, since ClickHouse server is open source under the Apache License 2.0. The main drivers are the number of shards and replicas, how many tables and dependent objects must be verified, the size of the backup and restore job, the amount of spare hardware you need for restore tests, and how many change windows the work takes. Some costs are one-time, such as writing the rollback runbook and proving a restore on a spare cluster. Others recur, such as repeating restore tests for each new release line. When you request a quote, ask for the written scope, the assumed versions, what is tested and how, who provides the spare environment, and what evidence is delivered at the end.
How long does a ClickHouse rollback take?
Duration depends on data volume, cluster topology, network and storage throughput, and how many verification steps your queries and dashboards require. Restore time scales with the data you must bring back, and the time spent validating results often exceeds the restore itself. No general duration applies to every cluster, so ask for a timeline based on your measured data size and a rehearsal on a representative copy. Confirm the expected duration of a full restore from your chosen backup target, the time needed to resync replicas after restore, and the window your application owners can tolerate for read or write interruption.
What does our team need to provide for a rollback?
Plan for access and decisions before the upgrade starts. You need read and write access to the backup location, such as local disk, S3-compatible storage or Azure Blob Storage, and credentials that can run BACKUP and RESTORE. You also need a spare cluster or isolated environment with enough capacity to rehearse restores, change windows agreed with application owners, and named owners for the go or no-go decision, the database operations, and business validation of query results. Confirm who can approve a rollback at each stage and who can reach the on-call team outside working hours.
Can our own team run a ClickHouse rollback without a partner?
Yes, if your operators already have hands-on ClickHouse experience, a documented runbook, and a restore that has been tested on a spare cluster. The tasks that need care are checking version compatibility, taking consistent backups, stopping and restarting replicas in the right order, and verifying data after restore. Outside help is most useful when you have no one who has run a restore before, when the upgrade crosses several release lines, when you must produce audit evidence for an assessment, or when the rollback has to run during a live incident. Ask any partner for a written scope that defines what they run, what you run, and what you own afterward.
What should we do if a restore test fails?
Stop the rollback plan and treat the failure as a finding. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed, so a failed test is the early warning that the process has a gap. Record the exact error, the source and target versions, the backup method and the storage target, then check whether the cause is a version incompatibility, a missing object, a permission problem, an incomplete backup or a configuration difference. Keep the production upgrade paused, fix the cause on the spare cluster, and repeat the test until the restore and validation pass. Confirm who signs off on resuming the change.
How DeployOpen scopes and runs a ClickHouse upgrade with a rollback plan
DeployOpen plans and runs self-managed ClickHouse upgrades inside your own infrastructure, starting from a written scope that names what can and cannot be rolled back. We build the backup, restore test and rollback steps before the upgrade window, then document and hand them over so your team can run the plan without us.
How we scope
- Inventory every cluster, shard, replica and Keeper node with its current ClickHouse version and configuration.
- Record which features, table engines and settings are already in use, since they determine whether a downgrade is possible.
- Agree in writing the target release, the rollback trigger conditions and the restore point objective for each cluster.
How we build
- Take and verify backups with BACKUP before the upgrade, then restore them to a spare cluster as a test.
- Rehearse the upgrade and rollback order on a pre-production environment with representative data and queries.
- Review the changelog for backward-incompatible changes and test affected queries, including analyzer-related ones, before the change window.
How we hand over
- Deliver a written runbook covering the upgrade sequence, go or no-go checks, rollback steps and post-upgrade verification.
- Document the backup locations, restore procedure and the evidence from the restore test for your change records.
- Walk your operators through the runbook and the rollback decision points so your team can execute it independently.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer if your team wants DeployOpen to stay involved.
- Ask any quote to separate the one-time upgrade and rollback planning from recurring support and restore testing.
