Keeping self-managed ClickHouse available with ClickHouse Keeper, ReplicatedMergeTree, shards and Distributed tables, and recovering replicas when nodes fail.
Building a production Keeper ensemble for ClickHouse replication: why three or five nodes, dedicated hosts or disks, server IDs and hostnames, log and snapshot paths, secure Raft traffic, and async replication on current versions.
Replacing an existing ZooKeeper ensemble with ClickHouse Keeper: planning the maintenance window, pausing background tasks, converting snapshots with clickhouse-keeper-converter, verifying metadata and rolling back if needed.
Creating replicated tables on self-managed ClickHouse: Keeper paths with shard and replica macros, insert quorum for durability, digest ACLs, and checking replication health in system tables.
When a single server is too small or too slow: defining clusters, choosing a sharding key, creating Distributed tables with ON CLUSTER, routing inserts, and the trade-off against scaling one server up.
The documented ways to spread data onto a new shard when ClickHouse will not rebalance on its own: weighted writes, moving partitions, or copying with INSERT SELECT, and how to plan each.
Replicating schema changes across a ClickHouse cluster with the Replicated database engine instead of ON CLUSTER DDL: how it works with Keeper, adding replicas, and the limits to know before adopting it.
Applying CREATE, ALTER and DROP across all ClickHouse nodes with ON CLUSTER: the distributed DDL queue in Keeper, timeouts, failed hosts, and safe practices for schema changes in production.
Letting new ClickHouse nodes join a cluster definition through Keeper instead of editing remote_servers on every host: configuration, observer nodes, and where it helps on Kubernetes and autoscaled fleets.
What happens when a replica restarts after a failure or loses its data: automatic resync from healthy replicas, the safety check that blocks startup, the force_restore_data flag, and rebuilding a replica from scratch.
Changing Keeper ensemble membership without a full restart using dynamic reconfiguration: enabling reconfig, keeping quorum while replacing hosts, and why hostnames beat IP addresses.
Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.