Zero-copy replication in ClickHouse: risks and alternatives

Should replicated ClickHouse tables on S3 use zero-copy replication?

Zero-copy replication on S3 is off by default and not recommended for production. A team storing large MergeTree datasets on object storage may be tempted by it because, in principle, replicas point at the same stored parts instead of each keeping a full copy, which can cut object storage capacity and the upload work for every replica. In plain terms, it trades duplicate data for shared data. Whether that trade is ever acceptable depends on the ClickHouse version in use and on which failure modes your team is willing to accept. Confirm the current status of the feature and its settings in the release notes and documentation for your exact version before you plan around it.

The risk sits in what sharing means. If replicas depend on the same underlying objects, a damaging operation such as an accidental deletion, a bad merge or corruption can affect every replica at once, so a healthy replica may not be a safe source of recovery. ClickHouse's backup documentation makes the same point about replication in general: it does not protect against human error or corruption. Ask whether the feature's recovery behavior has been tested for your failure scenarios, and whether your team has a runbook for restoring from something other than a surviving replica. Write these answers down before a production decision is made, because they determine whether the storage saving is worth the exposure.

For most teams the safer path is standard ReplicatedMergeTree on object storage, where each replica keeps its own copy of the data and ClickHouse Keeper coordinates the replicas, with a recovered replica syncing missing data from a healthy one. That costs more object storage, so size the bucket and cache with that in mind. Pair it with BACKUP and RESTORE to storage that is separate from the cluster, and run restore tests on a spare cluster on a schedule. Our replication guide reference covers the replica model that this path depends on.

Replication options for tables on object storage

This table compares zero-copy replication on S3 with the supported pattern of ReplicatedMergeTree tables where each replica keeps its own copy, plus a non-replicated S3 disk protected by backups. Read each row as a trade between storage cost, what happens when a node or bucket fails, and how much operational risk you accept. The main takeaway is that the storage saving from zero-copy does not come with the failure protection most production teams expect, so confirm current defaults and settings in the ClickHouse replication and S3 disk documentation before choosing.

Option	What it does	Storage cost profile	Failure behavior	When it fits
Zero-copy replication on S3	Replicas point at one shared set of data parts in the bucket instead of each keeping a copy	Lowest object storage use, since parts are not duplicated per replica	Shared parts become a single point of failure, and the feature is not recommended for production	Test or evaluation only, if at all, after reviewing current ClickHouse guidance
ReplicatedMergeTree with a copy per replica	Each replica stores its own full copy of the data, coordinated by ClickHouse Keeper	Highest object storage use, roughly one copy per replica plus overhead	A failed replica resyncs missing data from a healthy replica without manual steps	Production default for replicated tables that must stay available
Non-replicated S3 disk with backups	One copy of data in the bucket, protected by scheduled BACKUP to separate storage	Lower than full replication, plus the cost of backup storage and restore tests	Losing the disk or bucket means restoring from backup, so recovery time depends on backup freshness	Analytics where some data loss window is acceptable and restores are tested regularly
Replicated tables on local disks with tiered object storage	Hot data sits on replicated local volumes, older data moves to S3 under a storage policy	Mixed: replica copies on local disks plus one object storage footprint for cold data	Replica failover still applies to local parts, and object storage follows its own durability model	Large retention where recent data needs fast replicated access
Backups as the recovery layer	BACKUP and RESTORE capture tables, partitions or databases to local disk, S3-compatible or Azure storage	Recurring cost for backup storage and the compute to run and test restores	Replication does not protect against human error or corruption, so backups carry that role	Every production cluster, whatever replication option is chosen

Risks that keep zero-copy replication out of production

Each risk below shows how zero-copy replication changes what a failure, a bad delete or a recovery does to your data. Together they explain why the feature is off by default and why replication should never be your only protection.

Shared data across replicas

With zero-copy replication, replicas of a ReplicatedMergeTree table point at the same stored data parts instead of each keeping a full copy. A fault in those shared parts, or a bad delete run on one replica, can reach every replica that uses them. Ask how your version handles shared parts during merges, drops and mutations, and whether one bad operation can reach every replica.

Bucket availability and rules

The replicas' data sits in one object storage bucket, so bucket availability, access policy, lifecycle rules and versioning become part of the cluster's availability and durability design. A lifecycle rule that expires objects, a credential rotation that cuts access, or an outage in one bucket can stall or damage every replica at once. Confirm with your storage provider which bucket rules apply, how deletions and object expiry behave, and what replicas do during a bucket outage.

Replica recovery behavior

A replica that was down normally catches up by fetching missing data from a healthy replica, and the replication guide describes this sync as automatic. When replicas share stored objects, recovery depends on how the cluster reconciles those shared parts, a step many teams never rehearse. Test on a non-production cluster: stop one replica, write data, bring it back, and confirm that row counts and query results match the healthy replica. Rehearse recovery before you rely on it.

Replication is not a backup

Replication keeps copies of data available when a node fails, but it does not protect against human error or corruption, because a bad statement is replicated too. The backup documentation states this directly. Zero-copy replication weakens the position further, since every replica depends on shared objects. Send BACKUP jobs to storage outside the cluster's bucket and account, with separate credentials, so one deletion cannot remove both live data and backups. Replication is not a backup.

Restore tests on spare clusters

The backup documentation warns that a backup never test-restored will probably not restore when you need it, and recommends regular restore tests on spare clusters. For zero-copy setups, each test should restore into a separate cluster, then check table schemas, row counts and sample query results. Ask who owns each test, how often it runs and where results are stored, so the evidence exists when an auditor or an incident review asks for it.

Feature status in your version

Zero-copy replication is off by default and not recommended for production, so enabling it is a deliberate exception that needs a written justification. The security policy lists 26.9, 26.8, 26.7 and 26.3 as supported with security updates, and all other versions are unsupported. Before any decision, confirm the feature's status, known issues and upgrade behavior for your exact release in the changelog, and confirm your version is on the supported list.

Which supported alternatives fit replicated tables on object storage

The supported path for replicated ClickHouse tables is ReplicatedMergeTree on local or block-storage disks, with each replica holding a full copy of its data and ClickHouse Keeper coordinating replication metadata. Keeper replaces ZooKeeper, and the replication guide recommends at least three Keeper nodes on dedicated hosts in production, each with about 4 GB of RAM. This is the design to plan around when you need high availability. Replication handles node loss well: a recovered replica automatically syncs missing data from a healthy replica without manual intervention. Confirm the Keeper topology and disk layout with your own sizing tests before you commit to it, since the replication guide sets a starting point, not a capacity figure for your workload.

If you need object storage, keep it for tiered data and treat each replica's relationship to that storage as a deliberate design decision. Use a storage policy that moves older parts to an S3-compatible or Azure disk, and confirm with your own tests how each replica reads and caches those parts, what the failure behavior is when the bucket is unavailable, and which queries tolerate the added latency. Object storage is a capacity and retention layer in this model, not a substitute for a second full copy of the data. Review the ClickHouse replication guide for the current supported configuration before you settle the replica model.

Recovery from deletion, corruption or a bad migration is a separate control. Use BACKUP and RESTORE to write full and incremental backups of tables, partitions or databases to S3-compatible or Azure Blob Storage, because replication copies a mistake to every replica as quickly as it copies good data. Schedule backups to a location outside the failure domain of the cluster, and run restore tests on a spare cluster on a regular cadence. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed. Ask your team to record the restore time they actually measure, since that figure, not the backup job's success status, is what your recovery plan should rely on.

Setting up replicated tables on object storage without zero-copy

This sequence gives you a replicated ClickHouse setup on object storage that does not depend on zero-copy replication, with each layer checked before the next is built. The output is a supported version, Keeper quorum, replicated tables, a tested backup target and a restore result you have seen succeed.

Confirm the ClickHouse version and support status
Your platform or database lead checks the version you intend to run against the ClickHouse security policy and the release list on GitHub before anything is installed. Choose a stable or LTS line that is still supported, and read the changelog for backward-incompatible changes, because the production FAQ notes that minor releases can include them. Record the exact version in your change record. Done when every ClickHouse node in the plan runs the same supported build and the version is written down with its support window.
Define the storage policy and object storage disks
The storage engineer defines an S3-compatible disk and a storage policy that points replicated tables at it, with each replica keeping its own copy of the data. Confirm with your object storage provider that the bucket, credentials, endpoint and lifecycle rules match the policy, and keep bucket-level deletion and expiry rules out of ClickHouse-managed paths. Validate the configuration by restarting one node and querying system.disks and system.storage_policies to confirm the disk is visible with the expected policy. Done when each node reports the same disks and policy.
Deploy at least three Keeper nodes on dedicated hosts
The platform team deploys ClickHouse Keeper as a quorum of at least three nodes on dedicated hosts, following the replication guide, which suggests about 4 GB of RAM per Keeper node for production. Keep Keeper off the ClickHouse data hosts so disk or memory pressure on a data node cannot break coordination. Check the quorum with the Keeper four-letter commands or the ruok and mntr responses, and confirm one leader and the rest followers. Done when all three nodes answer and a test znode written through one node is readable from another.
Create ReplicatedMergeTree tables on the storage policy
The database engineer creates ReplicatedMergeTree tables with a unique ZooKeeper path per table and a replica name per node, and attaches them to the storage policy from the earlier step. Use the same schema on every replica and avoid creating tables directly on one node only. Insert a known set of rows through one replica and check system.replicas on the others for is_readonly = 0, a growing absolute_delay that returns to zero, and an empty queue_size once sync completes. Done when every replica reports the same row count and a healthy replication status.
Verify replica recovery after a controlled node stop
The operations team stops one replica in a maintenance window, writes more rows through a healthy replica, then starts the stopped node. According to the replication guide, the recovered replica syncs missing data from a healthy replica without manual steps, so the test checks that behavior in your own environment. Compare row counts and checksums on the affected table across replicas after sync. Done when the restarted node matches the others and system.replicas shows no backlog. Record the time it took in your own runbook rather than assuming a figure.
Configure the backup target and schedule
The backup owner sets up BACKUP to a separate target, which can be local disk, S3-compatible storage or Azure Blob Storage, in an account or bucket that replicas cannot delete from by default. Write a schedule that runs full and incremental backups of the databases or tables that matter, and record each backup name in a catalog. Replication does not protect against human error or corruption, so a bad DROP replicates everywhere. Done when one scheduled backup completes and the backup name, size and location appear in your catalog and monitoring.
Run a restore test on a spare cluster
The recovery team restores the latest backup into a spare ClickHouse cluster that matches the production version, storage policy and Keeper layout as closely as practical. Use RESTORE against the same table names in a separate database, then compare row counts, schema and a sample of queries with production. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed, so repeat this on a schedule you can defend to auditors. Done when the restored data matches the source within the agreed check and the result is written into the recovery log.

Security and operational controls for replicated object storage

Replicated tables on object storage concentrate risk in a few places: the bucket credentials every replica holds, the paths that carry data and coordination traffic, the backup copies that must stay separate, and the storage policy that decides where data lives. The controls below address those points, and each one is a decision to make and verify in your own environment before production.

Scope bucket credentials to least privilege

Every ClickHouse replica that uses an S3 disk holds credentials for the bucket, so a leaked key exposes every table stored there. Create a dedicated principal for the disk, grant only the object actions the disk needs on the prefixes it uses, and deny deletion of other tenants' or backup prefixes. Confirm how your object store scopes keys by prefix, and rotate keys through your secrets process rather than editing server configuration by hand. Access control on the database side is covered by users, roles, row policies and settings profiles, and the access control documentation recommends restricting the default user, which starts with all permissions.

Encrypt traffic to storage and Keeper

Replication sends data to object storage and coordination messages to ClickHouse Keeper, so both paths need protection in transit. Use TLS for connections to the object store where your endpoint supports it, and confirm that certificate validation is enabled rather than skipped. Configure TLS between replicas and Keeper nodes as well. Test each connection with the certificate chain your organization actually issues, because a path that works in staging can still fail or fall back to plaintext in production.

Separate backup and replica credentials

A replica credential that can write, overwrite or delete objects is also a route to destroying backups stored in the same bucket. Keep the principal that replicas use separate from the principal that BACKUP and RESTORE use, and give the backup principal its own bucket or prefix with restricted delete rights. Replication alone does not protect against human error or corruption, so backups must remain independent of the replicated data path. Test restores with the backup credentials to confirm they still work.

Monitor replica lag and Keeper health

A replica that falls behind or loses its Keeper session can serve stale data without an obvious error. Track replication queue length, replica delay and read-only status for each replicated table, along with Keeper member availability and latency. A recovered replica syncs missing data from a healthy replica automatically, so alert on replicas that stay behind rather than assuming recovery. Set thresholds with your own team, using the metrics your monitoring stack collects from the server.

Use external authentication for operators

Operator and service logins should not depend on local database passwords that are copied between teams. ClickHouse supports LDAP, Kerberos, SSL X.509 certificates, an HTTP authenticator and JWT as external authenticators, so you can tie administrative access to the identity system you already run. Decide which authenticator each role uses, and confirm that the default user is restricted before any operator connects. Review group mappings whenever staff change roles, since storage and backup rights often follow the same groups.

Control changes to storage policy

A storage policy controls which disk receives new parts and which volumes move them, so a small edit can send data to the wrong bucket or stop replication. Keep policy files in version control, require review for each change, and test edits on a pre-production cluster with representative data before rollout. Record which version is running, because the production FAQ recommends reading the changelog before upgrading. Plan each change with a rollback step so an operator can restore the previous policy without guessing.

Limits to confirm by environment

In a cloud account, the main limits come from the network paths and managed services around the cluster. Object storage endpoints, private connectivity and egress rules decide whether replicas and backup targets can reach the buckets they depend on, and cloud providers change default security settings over time. Treat any cloud-specific default as something to verify against the account you actually run.

Supported versions: Check the ClickHouse security policy for the versions currently receiving security updates, and match your cluster's release line to that list before you design around it.
Release channel: Decide whether the cluster follows stable releases or LTS releases, and record who approves a move between channels.
Object storage endpoints: Confirm which bucket endpoints your VPC, firewall and private link rules allow from every replica and Keeper host.
Backup targets: Verify that the backup bucket or container is reachable from the cluster and from the restore environment, including its credentials and retention settings.
Managed defaults: Check the default bucket encryption, lifecycle and versioning rules, since they can interact with ClickHouse deletes and merges.

Before you commit, confirm each setting and default in the version of the ClickHouse documentation that matches your release.

Questions teams ask before replicating ClickHouse on object storage

Does zero-copy replication actually save money on object storage?

It can reduce stored bytes, but the saving is smaller and less certain than it looks, and it depends on your data volume, replica count and retention. Zero-copy replication on S3 is off by default and not recommended for production, so the saving should be weighed against the operational risk it carries. The main cost drivers are the volume of data kept per replica, the object storage request and egress pricing of your provider, the local cache size, and the hardware for Keeper nodes. Ask the provider and your architect to model all of these with your own retention and ingest rates, and to show the cost of the supported alternatives beside it.

How long does a design review and proof of concept take?

The length depends on how many tables use replication, how much data and retention you need to test, and whether your storage and network are already defined. A review usually covers the storage policy, replica layout, Keeper placement, backup approach and failure scenarios. A proof of concept needs representative data volumes and queries so that you can measure restore and merge behavior. Ask for the phases, the environment each phase requires, the deliverables and the decision points in a written scope before agreeing to any timeline.

What does the work need from our own staff?

Expect to supply people who know your storage platform, network, identity and change process, and who can approve changes to production. You will also need someone who owns the ClickHouse cluster after handover, because replica recovery, upgrades and backup restore tests are recurring tasks. Name the staff, their hours per week during each phase, and who makes the final design decision. Ask the partner to state which of these tasks they will perform and which they expect your team to run after go-live.

Can our team design the replication setup without outside help?

Yes, if your engineers have operated distributed databases and can test failure scenarios in a non-production environment. The ClickHouse replication guide describes ReplicatedMergeTree tables coordinated by ClickHouse Keeper, and a recovered replica syncs missing data from a healthy replica on its own. The harder parts are sizing Keeper, choosing storage policies, validating backups with restore tests, and planning upgrades across replicas. An outside review is most useful for checking the design against your failure modes before you commit to it.

What should we ask for in a written scope?

A written scope should name the ClickHouse version and its support status at the time of build, the storage backends and their configuration, the number of shards and replicas, and the Keeper topology. It should also list the backup method, the restore test plan, the monitoring you will receive, the acceptance criteria, and what is excluded. Ask for each deliverable to be listed with an owner, a date or milestone, and a sign-off step. Require a statement of what happens if a test fails and how changes are handled.

How do we decide between replicas, tiering and backups?

Treat them as different controls. Replicas keep the cluster available when a node fails, but they copy mistakes such as accidental deletes, and the ClickHouse backup documentation notes that replication alone does not protect against human error or corruption. Tiering moves older data to cheaper storage to control cost, and it does not protect data on its own. Backups with BACKUP and RESTORE, tested regularly on a spare cluster, protect against deletion and corruption. Decide each one against a stated recovery objective and a cost limit, and confirm the design with a restore test.

Is zero-copy replication supported for production use?

No. ClickHouse keeps zero-copy replication on S3 off by default and does not recommend it for production, so a design built on it would rely on a setting the project advises against. Before relying on any object storage replication pattern, check the current ClickHouse production guidance and the security policy for supported versions. Confirm in writing with your partner which version you will run and which replication method it will use.

How DeployOpen works on this

DeployOpen plans, implements, tests and documents self-managed ClickHouse inside your own infrastructure, so the replication design is decided against your data, storage and recovery requirements. We start by checking whether a zero-copy design on object storage is needed at all, and we build the supported alternative when it is not.

How we scope

Engagements are fixed-scope, with the replication design, deliverables and timeline agreed in writing before work starts.
The scope records which tables, retention periods and storage backends are in question for Zero-copy replication in self-managed ClickHouse.
Ask for the scope to state the ClickHouse version and Keeper layout you will run, since these decide which replication options apply.

How we build

We build replicated tables on ReplicatedMergeTree with ClickHouse Keeper, following the replication guide's recommended node count and host placement.
Object storage disks and storage policies are configured and tested against failure scenarios, not only normal writes and reads.
Each replica is checked for automatic resync after a simulated node loss, with the results recorded in the build log.

How we hand over

You receive the written design, the configuration files, and a record of which replication path was chosen and why.
Restore tests are run on a spare cluster, since the backup documentation warns that untested backups will probably not restore when needed.
Operators get a runbook covering replica recovery, Keeper health checks and the upgrade process for your supported version line.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover.
Retainers can cover reviewing changelogs before each stable or LTS upgrade, since minor backward-incompatible changes can occur.
Confirm with DeployOpen which support window and response arrangements apply, as these are set in the retainer agreement.

Zero-copy replication in self-managed ClickHouse