Recovering a failed or lost ClickHouse replica

How a ClickHouse replica recovers after a failure

A recovered ReplicatedMergeTree replica normally syncs its missing data from a healthy replica without manual intervention, according to the ClickHouse replication guide. If your node restarted or lost its disk, the first question is whether the cluster heals itself, and in most cases it does. Manual steps apply only in specific cases, such as when the differences between replicas are too large for automatic failover, which the replication guide describes as semi-automatic. Before you plan any recovery runbook, confirm with your own test cluster how long your replicas take to catch up on your data volumes.

Two components make this work. Replicated tables use the ReplicatedMergeTree engine to write parts of data to every replica in a shard, and ClickHouse Keeper coordinates those replicas by holding the shared replication metadata and log. Each replica reads that log to find what it is missing, so the Keeper ensemble is what a replica needs to reach before it can catch up. Check Keeper health and quorum as part of any incident, because a replica cannot resync if its coordination service is unavailable. The replication guide recommends at least three Keeper nodes in production, and the replication documentation covers the configuration details.

Replication is not backup. The backup documentation states that replication alone does not protect against human error or corruption, because a bad DROP or DELETE is copied to every replica. A replica that resyncs from a healthy peer also inherits any corruption that peer carries. Recovery restores redundancy; it does not restore data to an earlier point in time. Keep BACKUP and RESTORE jobs running to separate storage, and test restores on a spare cluster, since a backup that has never been restored may not restore when you need it.

What happens automatically when a replica rejoins

When a replica comes back after a restart or network loss, it does not need an operator to tell it what it missed. The replica reads its replication state from ClickHouse Keeper, compares it with the parts that healthy replicas hold for each ReplicatedMergeTree table, and fetches the parts it lacks from a healthy peer. According to the ClickHouse replication guide, this resync happens without manual intervention. The guide also describes failover as automatic when the difference between replicas is small and semi-automatic when the data has diverged too far for the replica to catch up on its own. In the semi-automatic case, the replica waits for an operator decision, so your runbook needs a named owner who can make that call.

After a rejoin, check the replication health views before you consider the incident closed. The system.replicas table shows whether each replica is readable, its queue length, and any exception recorded against the table, while system.replication_queue lets you see which fetches are pending or failing. Confirm that the queue drains, that absolute_delay returns to a small value on the recovered node, and that row counts or checksums for recent partitions match across replicas. Also check the server log for repeated fetch errors, since a replica can look present while it keeps retrying a part that will never arrive.

The exact resync behavior, the size of difference that counts as small, and the conditions that trigger the semi-automatic path can change between versions. Confirm them against the replication documentation for the release you run, and test a rejoin on a non-production cluster with a deliberately stopped replica before you rely on the timing in your own recovery plan. Keep your supported version list in view, since the ClickHouse security policy determines which releases still receive fixes and therefore which replication behavior you can expect to keep.

Choosing a recovery path by failure type

A replica that restarts after a crash, a host reboot or a network interruption usually still has its data parts on disk. ClickHouse compares its local state with the replication log coordinated by ClickHouse Keeper, fetches the parts it missed from a healthy replica and rejoins the shard. For small gaps this happens without operator action, so queries keep running on the other replicas while the restarted one catches up.

Expected behavior: The replica replays its replication queue and fetches missing parts until its delay returns to near zero.
First check: Confirm the server process started cleanly and that the replica can reach the Keeper ensemble.
Replication status: Query system.replicas for the affected tables and read the queue size, absolute delay and last exception fields.
Keeper health: If several replicas report session problems at the same time, check the Keeper ensemble before you touch any data.
When to escalate: Escalate if the queue does not drain or the same exception returns after each restart.

Before you close the incident, confirm that row counts and part lists on the restarted replica match a healthy peer.

When a replica's data diverges from its peers by more than the automatic resync can reconcile, ClickHouse does not choose a copy by itself. The replication guide describes failover as automatic for small differences and semi-automatic when the data differs too much. The startup safety check blocks the replica from starting, which protects the shard from a copy that may be wrong. The force_restore_data flag overrides that check, so use it only after you have decided which copy is authoritative.

Expected behavior: The affected table refuses to start or stays read-only until an operator intervenes.
First check: Read the server log for the startup error, then compare part counts and checksums across the replicas.
Choose the source of truth: Identify the replica with the most complete and verified data before you change anything.
Use `force_restore_data` sparingly: Set the flag only on the replica you plan to rebuild, and only after you have copied or backed up its current local state.
Ownership: Many teams bring in DeployOpen here, because a wrong choice can discard data the shard still needs.

Record which replica you chose as the source and the reason for that choice before you remove any local data.

Why the startup safety check blocks a replica

A ClickHouse replica can refuse to start when its local state does not match what ClickHouse Keeper records for its replicated tables. The check exists to stop a node from quietly rejoining a shard with data that may have diverged, been lost, or been left over from an earlier state. In practice, the startup refusal is a request for a human decision: the server is telling you that automatic resync may not be the right action on its own, and that someone should confirm what happened before data is rewritten.

When the check fails, the operator sees the problem in the server log rather than in a running replica. Replicated tables stay unavailable for writes or reads until the condition is resolved, and the log entries name the table and the mismatch. Exact messages change between releases, so confirm the wording against the version you run before writing alerts or runbooks around it. Restarting the node repeatedly without reading the log is risky, because each attempt can repeat work on the same local parts and hide the evidence you need, such as which replica held the most recent data and whether the local files were damaged.

The decision belongs to whoever owns the data. Before any forced recovery, record the replica state in the system tables for each replica, identify a healthy source, and decide whether the local copy is worth keeping. The force_restore_data flag overrides the protection and should be used only after that review, then removed once the replica has caught up. Plan this decision as part of your runbook, and test it on a non-production cluster first so the team knows what the logs look like before an incident.

Using force_restore_data safely

This sequence confirms that the replica is the right one to recover, sets the flag only after the surrounding state is verified, and ends with a replica that has rejoined its shard and reports healthy replication. It keeps the healthy replicas as the source of truth throughout.

Confirm the replica is the one to recover
Before touching any setting, the operator checks which node has lost its data or is stuck in a failed state, and confirms that the other replicas of the same shard hold the current data. Compare row counts and the newest partitions on the suspect replica against a healthy peer, and check the replica's logs for the startup error that blocked it. This step is complete when the team can name the damaged host and the healthy peers in writing, and has agreed that no healthy replica is involved in the recovery.
Stop the damaged replica cleanly
Stop the ClickHouse server process on the damaged host so that no inserts, merges or fetches run against its local data while the recovery is prepared. Record the host, the shard and the replica name, because these values must match the table definitions that are restored. This step is done when the process is confirmed stopped on that host and the other replicas still answer queries.
Verify Keeper and a healthy replica
Confirm that the ClickHouse Keeper ensemble has a quorum of available nodes, since the recovered replica reads its replication state from Keeper, and that at least one healthy replica of the same shard is reachable and serving the tables. Check the Keeper status on each node and a basic query on the healthy replica. The step is complete when both checks pass, because a recovery started without them can leave the replica with no reliable source to copy from.
Set force_restore_data to restore local state
Create the flag file named force_restore_data in the ClickHouse data directory on the damaged host, then start the server. The flag tells the server to restore its local replica state from what Keeper records, which allows it to start when the safety check would otherwise block it. Confirm the exact behavior of the flag for your ClickHouse version in the replication documentation before you set it, because the mechanics may differ between releases. This step is done when the server starts and writes its startup messages to the log.
Remove the flag after startup
Once the server is running, delete the force_restore_data flag file so that the next restart does not repeat the forced restore. Check that the file is gone and that the server restarts normally if you restart it in a controlled way. This step is complete when the flag is absent from the data directory and the replica stays up without it.
Watch the replica resync
Let the replica fetch the parts it is missing from a healthy peer, which the replication documentation describes as automatic for small differences. Watch the replication queue and the fetch activity in the system tables until the backlog drains. If the differences are large, expect the process to need operator attention rather than completing on its own, and plan for that with the team. This step is done when the queue is empty or shrinking steadily and no errors repeat.
Verify replication health end to end
Compare row counts, partition lists and checksums or sample queries between the recovered replica and its peers, and check the replication status table for readonly flags, delayed replicas and errors. Confirm that a test insert on one replica appears on the recovered replica. The step is complete when the replica matches its peers on the checks agreed in step one and replication reports no errors, and the result is recorded for the incident file.

Mistakes that turn a replica failure into data loss

Most replica failures in ClickHouse are recoverable, and the damage usually comes from an operator action taken in a hurry. The cards below cover the traps that turn a single failed node into lost or unrecoverable data, and the control that prevents each one. Read them before the next incident, not during it.

Using force_restore_data on the wrong node

The force_restore_data flag tells a replica to start even when its local state fails the safety check, which means it can come up with a different view of the data than its peers. Applied to a healthy replica or to the wrong host, it can discard the copy you needed. Before setting it, confirm which replica holds the most complete data, record that choice, and remove the flag once the node has resynced.

Treating replication as backup

Replication copies every insert and every mistake to the other replicas within seconds. A accidental DROP TABLE, a bad ALTER or a corrupted part spreads to the whole shard, so replication offers no way back. The ClickHouse backup documentation makes this point directly. Keep BACKUP and RESTORE jobs running to storage outside the cluster, and check that they cover every database and table you need.

Rebuilding without checking Keeper health

Replicas coordinate through ClickHouse Keeper, so a replica that resyncs while the ensemble has lost quorum can stall, retry or reach an inconsistent state. Before you wipe a replica or start a rebuild, confirm that a Keeper majority is reachable and that the other replicas of the shard are healthy and writable. Check the replication queue and the replica status tables first, and stop if they show errors you do not understand.

Reusing hostnames or macros incorrectly

ReplicatedMergeTree identifies each replica through its Keeper path and the shard and replica macros in its configuration. If a rebuilt node reuses another replica's name or macro values, or a new host takes an old identity without cleaning up its Keeper entries, the cluster can treat it as the wrong replica or refuse to register it. Keep macros unique per replica, use stable hostnames, and remove stale replica entries from Keeper before adding a replacement under the same name.

Skipping restore tests

A backup that has never been restored is a hypothesis, not a recovery plan. The ClickHouse backup documentation warns that an untested backup will probably not restore properly when you need it. Schedule restore tests to a spare cluster on a regular cadence, time how long the restore takes, and confirm that row counts and key queries match production. Record the results so the next restore has a baseline.

Restarting several replicas of one shard at once

A shard only stays readable and writable while enough of its replicas are available. Restarting two or more replicas of the same shard together, whether for patching, a node failure response or a maintenance script, removes the copies you would resync from and can leave the shard with no healthy source. Restart replicas one at a time, wait for each to finish syncing and show an empty replication queue, then move to the next. Apply the same rule during rolling upgrades.

Recovery options and when each fits

This table compares the five ways to bring a ClickHouse replica back, from automatic resync to restoring a backup. Read each row against the state of the failed node and the quorum of healthy replicas, then pick the least invasive option that matches what you can verify.

Recovery option	When it fits	What it requires from the team	Main risk
Automatic resync	The replica restarts cleanly and the differences from healthy replicas are small	Keeper is reachable, healthy replicas hold the data, and monitoring shows the replica catching up in system tables	Resync stalls or fails silently if Keeper or disks are unhealthy, so the replica looks up while it is behind
Semi-automatic recovery with manual review	Data differs too much for automatic failover, so an operator must confirm which replica is authoritative	Someone who can read replication queue and part state, compare replicas, and decide which copy to trust before forcing changes	Choosing the wrong source replica, or delaying recovery while the shard runs with less redundancy
force_restore_data flag	The server refuses to start because replica metadata in Keeper does not match local state, and you have confirmed the local data is safe to use	Ability to inspect the flag file, remove it after startup, and verify replication health afterward; change control for a production shard	Forcing a replica to reinitialize from the wrong state, or leaving the flag in place so the node restarts into an unexpected mode
Rebuilding from scratch	The local data is lost or corrupt and no usable copy exists on the node, while other replicas hold the shard	Clean disks and paths, the same replica configuration and macros, and time for the full copy from peers to complete	Duplicate or stale parts if old replica paths in Keeper are not removed first, and a long period with reduced redundancy
Restoring from BACKUP	Data was deleted or corrupted on all replicas, or every replica of the shard is lost	A tested backup set in local disk, S3-compatible or Azure Blob Storage, spare capacity and a documented restore sequence	A backup that was never test-restored may not restore properly, and data written after the backup point is not recovered

Security and access during recovery

Recovery work touches replicas, Keeper and the data itself, so the same access that fixes a node can also expose or damage it. The cards below cover who may run recovery steps, how traffic between replicas and Keeper is protected, how actions are recorded, and how the default account is closed off before a replica is rebuilt.

Limit who can run recovery commands

Replica recovery involves dropping and recreating tables, changing settings and, in some cases, setting force_restore_data. Grant those privileges only to a small operator role, not to application users or broad admin accounts. Check which roles hold ALTER, DROP and SYSTEM privileges on each node, and confirm that the people on call can perform the steps without holding permanent superuser rights.

Close off the default user

The default user starts with all permissions, so a replica that is accepting connections with that account is open to anyone who can reach the port. Before a rebuild, confirm that the default user is restricted or disabled and that operators sign in with named accounts. Test the restriction on a staging node first, because recovery scripts that depend on the default account will fail once it is locked down.

Authenticate operators externally

Where the organization already runs a directory, route operator sign-in through an external authenticator such as LDAP, Kerberos, SSL X.509 certificates or JWT rather than local passwords. This ties each recovery action to a person who can be removed from the directory on departure. Ask the platform team which method applies to the cluster and confirm that the operator account keeps working when the directory is degraded during an incident.

Encrypt replica and Keeper traffic

Replicas fetch parts from each other over the network, and Keeper carries the coordination state that decides which replica is healthy. Both paths should use TLS with certificates issued by your own authority, so a rogue host cannot impersonate a replica or Keeper node during resync. Verify the certificate chain and hostnames on every node after a rebuild, because a replica that comes back with the wrong certificate can fail to rejoin.

Restrict Keeper access

Keeper stores replication metadata for the whole cluster, so write access to its data directory or client port is effectively write access to the replicas. Limit which hosts can reach the Keeper client port, protect the snapshot and log directories with file permissions, and use the digest ACL options covered in the replication setup to stop unexpected clients from editing paths. Review these settings before a recovery, since a rebuilt replica needs the same paths and permissions to register again.

Record every recovery action

A recovery often involves several steps over hours, such as detaching parts, dropping a replica, and restarting with a flag set. Keep a timestamped record of who ran each command, on which host, and why, and store the query log and server logs off the node so they survive a rebuild. Check that query logging is enabled before the incident, because logs that were never written cannot be reviewed afterward.

Questions teams ask before recovering a ClickHouse replica

Does a ClickHouse replica recover on its own after a failure?

Often, yes. According to the ClickHouse replication guide, a recovered replica automatically syncs missing data from a healthy replica without manual intervention. Failover is automatic when the differences are small, and semi-automatic when the replica has diverged too far. The deciding factor is how much data the replica is missing and whether its local data is still intact. A replica that restarts with a few minutes of missed inserts is a different case from one whose disks were replaced, so the first step is always to check the state of each replica in the system tables before deciding on a path.

What makes a replica recovery costly or slow?

The main drivers are the amount of data the replica must copy, the bandwidth between nodes, and how many tables are affected. Replicas that were offline for a long time, or that lost their data directory entirely, must fetch far more parts than one that missed a short window. Other factors include the load on the healthy replicas during the copy, the Keeper cluster's responsiveness, and whether the team must stop and inspect the broken replica first. Ask your partner or internal team to estimate the copy volume from current table sizes and the outage length, and to test the path in a pre-production environment with representative data.

What does DeployOpen need from our team to recover a replica?

DeployOpen needs access to the affected cluster in your infrastructure, the ClickHouse and Keeper versions in use, the replication configuration for the affected tables, and the logs from the failed replica and its healthy peers. We also need a change window agreed in writing, a named owner on your side who can approve actions on production, and confirmation of the most recent backups and their restore status. Scope, deliverables and timelines are agreed in writing before work starts, so these inputs are reviewed as part of the scoping step.

Can our internal team recover a ClickHouse replica without a partner?

Yes, if the team has operators who understand ReplicatedMergeTree, Keeper paths and the replica state tables. The procedure involves checking replication health, deciding whether the replica can resync automatically, and, in rare cases, using the force_restore_data flag or rebuilding the replica from scratch. Each step carries risk on a live cluster, so teams without that experience usually benefit from a rehearsal on a spare cluster first. The cost of an internal team is mostly staff time and the opportunity cost of pulling engineers from other work, while a partner's cost follows the written scope.

How long does it take to recover a ClickHouse replica?

Duration depends on the volume of data to copy, the network between the replicas, the number of affected tables and how much diagnosis is needed before work starts. No duration figure is supplied for this topic, so treat any number you hear as unverified until it is based on your own table sizes and bandwidth. Ask for the duration to be tied to measured copy throughput in a test run and to a stated set of assumptions in the written scope.

Does recovering a replica replace our backups?

No. Replication protects against a single node failing, but it does not protect against human error, a bad DROP or DELETE that replicates to every copy, or corruption that spreads to all replicas. The ClickHouse backup documentation notes that replication alone does not protect against these events. Use BACKUP and RESTORE for full and incremental copies to local disk, S3-compatible storage or Azure Blob Storage, and test restores regularly on a spare cluster, since a backup that has never been restored will probably not restore properly when needed.

What is the force_restore_data flag and when should we use it?

The force_restore_data flag tells a replica to start even when ClickHouse detects that its data may not match the cluster, so it should be used only after the cause is understood. Using it on a replica with intact but out-of-date data can skip the safety check that exists to prevent silent divergence. Confirm the exact behavior for your ClickHouse version in the replication guide and agree the decision in writing with the data owner before anyone sets it.

How DeployOpen works on this

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including ClickHouse replicas running on your cloud accounts, data centers or restricted networks. For replica recovery, we start from your cluster's actual failure modes and recovery record, then agree the work in writing before anything changes on a live system.

How we scope

Review your ClickHouse version, Keeper ensemble, replica layout and whether backups are restorable today.
Agree in writing which recovery paths apply: automatic resync, guarded startup and rebuild from scratch.
Confirm who approves forced restores and how data loss is judged before any recovery step runs.
Engagements are fixed-scope, so deliverables and timelines are set in the written scope before work starts.

How we build

Reproduce a replica failure in a pre-production cluster with representative data before touching production.
Write runbooks for resync checks, the startup safety check, the force restore flag and full rebuilds.
Test each runbook against the restore path, including a restore from BACKUP onto a spare cluster.
Harden Keeper access and replica settings so recovery does not widen exposure in your environment.

How we hand over

Deliver the recovery runbooks, health checks and replication status queries your operators will use.
Walk your platform and on-call teams through each recovery path with the test results in front of them.
Document configuration, ownership, escalation paths and the evidence produced by each restore test.
Record open risks and the changes that would need a new recovery test, such as a version upgrade.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainers can cover periodic restore tests, review of Keeper health and new ClickHouse releases in your environment.
Ask any partner for the support scope, response arrangements and change process in the written quote.