Choosing ClickHouse disks, RAID and filesystems before the first node goes live
Storage layout is one of the decisions on a self-managed ClickHouse server that is hard to change later. Once data arrives, moving to a different RAID level, stripe size or filesystem means rebuilding nodes and copying large volumes of parts, which is slow and risky on a live cluster. ClickHouse publishes a short set of guidance in its OSS usage recommendations: RAID-10 with the far layout when budget allows, RAID-6 or RAID-50 in place of RAID-5 once you have more than four disks, a 1 MB stripe size because average writes are about 1 MB, and replication regardless of which RAID level you choose.
Teams need to settle this before the first node is racked or the first cloud volume is attached. The choice depends on what the hardware is for: write-heavy event or log ingestion, read-heavy dashboards, or a mix, and on whether your platform offers local SSD, local HDD or network-attached block storage. Ask your hardware or cloud team to confirm what each candidate disk type sustains for sequential writes and random reads, and whether the controller cache is battery or flash backed. Those answers decide whether the RAID guidance above is practical in your environment.
This section sets the decision frame for the rest of the page. The sections that follow compare SSD and HDD, walk through each RAID layout and stripe size, explain the ext4 and XFS choice, and cover the caveats for NFS. One point applies to every option: RAID protects against a failed disk, not against a bad delete, a faulty query or a corrupted part, so replication and tested backups are still required on top of the array.
SSD or HDD for ClickHouse data: which fits which workload
When dashboards, APIs or customer-facing features run many concurrent queries against recent data, query latency depends on how quickly ClickHouse can read the parts it needs while also keeping background merges moving. In this situation, SSD-backed storage is usually the starting point because random reads and merge I/O compete for the same device, and slow media stretches tail latency under load. The media choice interacts with concurrency: more simultaneous queries mean more outstanding reads, so a device that performs well for a single query may degrade quickly when the cluster is busy.
- Merge pressure: Frequent inserts create many small parts that must be merged, and merges write data while queries read it.
- Concurrency profile: Peak concurrent queries and their read volume matter more than average throughput.
- Cache effects: The share of hot data that fits in memory and the OS page cache reduces how often queries touch disk.
- Write amplification: Replication and merges multiply physical writes, which affects device endurance planning.
- Budget trade-off: Faster media costs more per terabyte, so keep only the hot data on it where your access pattern allows.
Before buying, benchmark with representative data volumes, your real insert rate and the query mix at peak concurrency, and record p95 and p99 latency alongside merge backlog.
When the dataset is large, retention is long and most queries scan wide ranges of data in large sequential reads, HDD capacity can be economical. ClickHouse stores data in columnar parts and reads contiguous column files, so throughput on sequential access often matters more than per-query random latency. The trade-off is that spinning disks handle concurrent random access poorly, so the number of simultaneous queries and merges becomes the limiting factor rather than raw capacity.
- Sequential access pattern: Range scans over time-ordered data suit HDD throughput better than point lookups.
- Merge scheduling: Merges are I/O heavy, so plan for merge windows and watch whether merges fall behind ingestion.
- Concurrency ceiling: Large numbers of parallel scans on spinning disks can saturate the device quickly.
- Capacity growth: Long retention often drives the decision toward HDD, with the cost difference growing as data volume rises.
- Tiering option: Keep recent, frequently queried data on faster media and move older parts to slower storage through your retention design.
Test a realistic long-range query against a full-size partition, and confirm with your vendor or storage team the expected sequential throughput and failure behavior of the drives you plan to use.
Many teams run ingestion, dashboards and ad hoc investigation on the same servers. In this case, the media decision is rarely all-or-nothing. A single pool of fast media can handle the mix, or a split layout can place recent data on SSD and older data on HDD. The right choice depends on how much of the query volume touches recent data and how much is historical analysis that can tolerate slower responses.
- Hot and cold split: Recent partitions carry most query traffic, so they benefit most from SSD.
- Ad hoc queries: Unpredictable scans across history can slow interactive queries on shared disks.
- Resource isolation: Separate workloads onto different replicas or clusters when one class of query affects the others.
- Operational simplicity: A single media type is easier to size, monitor and replace than two tiers.
- Growth path: Plan how the split will change as retention grows and query patterns shift.
Run your actual mixed workload for a representative period and measure how background merges, ingestion lag and interactive query latency move together before committing to a layout.
RAID layouts and stripe settings compared
This table compares the RAID layouts and stripe setting that ClickHouse's own usage recommendations address, so you can see how capacity, write behavior and rebuild exposure shift between options. Read each row against your disk count and budget, then confirm what your controller or software RAID supports. The main takeaway is that RAID-10 is the first choice when budget allows, and that RAID does not replace replication.
| Layout or setting | Trade-off | When it fits |
|---|---|---|
| RAID-10 (far layout) | Mirrored and striped, so usable capacity is about half of raw. Writes go to both halves of each mirror, and reads can be spread across copies. | The recommended choice when budget allows, especially for write-heavy event and log ingestion. |
| RAID-6 | Two parity disks, so usable capacity is raw minus two disks. Each write updates parity, which adds write overhead, and rebuilds read every surviving disk. | Dense HDD arrays where capacity matters more than write speed. ClickHouse recommends it over RAID-5 with more than four disks. |
| RAID-50 | Stripes across several RAID-5 groups, so it keeps more capacity than RAID-10 while limiting the size of each parity group. A single group failure can still lead to a long rebuild. | Larger disk counts where RAID-6 write cost is too high and ClickHouse's recommendation for RAID-5 alternatives applies. |
| RAID-5 | Single parity, so one disk can fail without data loss. A second failure during rebuild loses the array, and rebuild time grows with disk size. | ClickHouse advises avoiding RAID-5 with more than four disks. Confirm whether a small array is acceptable for your risk tolerance. |
| Stripe size of 1 MB | ClickHouse recommends a 1 MB stripe because average writes are about 1 MB, so each write maps to whole stripes instead of being split. | Default starting point for ClickHouse data volumes. Confirm your controller accepts this value, since some controllers cap stripe size. |
| Replication on top of RAID | RAID protects against disk failure only. It does not protect against deletion, corruption or site loss, so ClickHouse recommends replication regardless of RAID layout. | Every production layout. Pair RAID with ReplicatedMergeTree and tested backups. |
ext4 or XFS, and why NFS needs extra caution for ClickHouse data
Treat the choice between ext4 and XFS as a decision to confirm, not a default to copy. The starting point is the current ClickHouse OSS usage recommendations, which cover filesystem guidance alongside the RAID and stripe advice in this topic. Read them against the ClickHouse version you plan to run, then test the candidate filesystems on your own hardware with a representative ingest and merge workload. Measure sustained insert throughput, background merge behavior, and how long a full data directory takes to check and remount after an unclean shutdown. Record the mount options you intend to use, such as those set in /etc/fstab, so the test reflects production. Do not select a filesystem on the strength of a generic benchmark, because ClickHouse's write pattern, part sizes and merge load are specific to your data.
NFS needs more caution than local disks for ClickHouse data parts. ClickHouse writes immutable parts and merges them in the background, so it depends on the storage behaving predictably under concurrent reads, writes and file renames. Ask the storage team and the NFS vendor how locking is implemented and how it behaves when a client loses its connection. Measure latency under merge load, not only idle throughput, since slow metadata operations can delay inserts and merges. Confirm how the server reports consistency after a failover, and whether a stalled mount blocks the ClickHouse process. Run failure tests that pull the network path mid-merge and during a server restart before any production use. Treat any network-attached storage as unproven for ClickHouse until your own benchmarks and failure tests pass, and keep replication across independent servers in place regardless of the storage backend.
Why replication still matters on top of RAID
RAID keeps a server running when one disk fails, but it does nothing about failures above the disk layer. Replication, backups and restore tests cover the gaps that RAID leaves open on a self-managed ClickHouse node.
What RAID protects against
RAID keeps a volume online when a single disk fails, so the server can keep serving queries while the failed drive is replaced and the array rebuilds. That is the only failure class it is designed for. Before relying on it, confirm the controller model, the rebuild behavior and whether your team gets alerts when an array enters a degraded state. A degraded array that nobody notices is still a single point of risk.
Controller and host faults
A RAID controller, its cache battery, the backplane or the host itself can fail as a unit, taking every disk in the array offline at once. RAID does not help when the failure is above the disks. For each ClickHouse node, identify which hardware component is shared across its disks, and ask whether a second node holds the same data on separate hardware.
Replicas through ReplicatedMergeTree
ReplicatedMergeTree tables keep copies of data on more than one replica, coordinated by ClickHouse Keeper, which replaces ZooKeeper. This is how a host loss or controller fault becomes a recovery event rather than a data loss event. Plan at least three Keeper nodes on dedicated hosts for production, and confirm that each replica sits on separate hardware and, where needed, a separate rack or site.
Automatic sync after recovery
When a failed replica comes back, ClickHouse syncs the missing data from a healthy replica without manual intervention. Your operational work shifts to confirming that the replica rejoined, that its lag is closing and that Keeper is healthy. Build alerts on replication state so that a replica that stays behind is noticed before the next failure.
What replication does not protect
Replication copies mistakes as faithfully as it copies good data. A dropped table, a bad delete or corruption that spreads across replicas will exist on every copy. The ClickHouse backup documentation states that replication alone does not protect against human error or corruption, so a separate backup is required for those cases.
BACKUP, RESTORE and restore tests
BACKUP and RESTORE support full and incremental backups of tables, partitions or databases to local disk, S3-compatible storage or Azure Blob Storage. Store backups off the host that holds the data. The documentation warns that a backup never test-restored will probably not restore properly, so schedule restores onto a spare cluster and time them against your recovery objectives.
How to validate a storage layout before production load
This sequence checks each storage decision in order, from disk hardware through RAID, filesystem, benchmarks, failure drills and restore tests, so the team knows the layout holds before real data arrives.
Confirm disk types and controller support
Inventory every disk model on the planned servers and record whether each is SSD or HDD, along with the RAID controller model, its cache and battery or protection status, and whether it can run in pass-through or JBOD mode. Ask the hardware vendor to confirm that the controller supports the RAID level you intend to use and that the firmware is on a version the vendor supports. The step is done when the inventory is signed off by the platform owner and any unsupported combination has been replaced or ruled out.
Build the RAID layout and stripe size
Create the array on each server according to the chosen design. ClickHouse recommends RAID-10 with the far layout when budget allows, and RAID-6 or RAID-50 instead of RAID-5 once you have more than four disks. Set the stripe size to 1 MB, the value ClickHouse's OSS usage recommendations give for average writes of about 1 MB. Record the exact controller commands or settings used. The step is done when the array reports the expected capacity, layout and stripe size on every node, and the configuration is written into your build documentation.
Choose and format the filesystem
Select ext4 or XFS according to your operating system standard and your team's operating experience, then format the array with the chosen filesystem and the block size your platform team uses. Keep the same filesystem and options on every node so that replicas behave alike. The step is done when the filesystem is created on every node and a short sample write and read succeeds with the expected performance characteristics, with the choice recorded alongside the reason.
Mount and check options
Mount the filesystem at the path where ClickHouse will store its data, using options your storage and security reviewers have agreed on, and add it to the fstab or equivalent so it survives reboots. Confirm that the mount is local block storage and not a network share. If you are considering NFS, confirm with the ClickHouse documentation and your storage vendor how that protocol behaves for your workload before using it for data. The step is done when a reboot returns the same mount, ownership and permissions, and the ClickHouse service user can write to the path.
Run benchmarks with representative data
Load a representative sample of your event, log or telemetry data into a pre-production cluster and run the ingestion and query workloads you expect in production, using the same concurrency where you can. Measure sustained insert throughput, merge behavior and query latency, and watch disk utilization and I/O wait during the runs. Compare the results against the requirements agreed with the business owners. The step is done when results from at least one full representative run are recorded and any shortfall has a documented decision to fix the layout or accept the limit.
Simulate a disk failure and rebuild
With the cluster loaded, pull or fail one disk in a test server, following the controller's procedure, and observe how the array degrades, how the server and ClickHouse respond, and how long the rebuild takes on your hardware. Record the rebuild duration and the performance impact on queries and inserts during the rebuild. Then replace the disk and confirm the array returns to its optimal state. The step is done when the degraded behavior and the rebuild are documented and the platform team accepts them as the operating baseline.
Run a replica recovery test
Stop one replica node that holds ReplicatedMergeTree data, keep writes flowing to the surviving replicas, then bring the node back. Confirm that the recovered replica syncs its missing data from a healthy replica without manual intervention, as described in the ClickHouse replication guide, and compare row counts and checksums across replicas. Keep in mind that replication protects against hardware loss but does not protect against deletes or corruption that replicate. The step is done when the replicas match and the recovery steps are written into the runbook.
Run a backup restore test
Take a full backup with BACKUP to the target you have chosen, such as local disk, S3-compatible storage or Azure Blob Storage, then restore it into a spare cluster with RESTORE and compare the restored tables against the source. The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed, so schedule this as a recurring exercise. The step is done when the restored data matches the source and the restore procedure, timing observed and owner are recorded for the production runbook.
Operational risks of the storage layout
A RAID layout is only as safe as the way it is watched, rebuilt and secured after launch. The cards below cover the operational decisions that follow the layout choice, and the items to confirm with your hardware vendor and security team before the servers go into production.
Rebuild windows and degraded performance
When a disk fails, the array rebuilds onto a spare or replacement, and reads and writes slow down while it runs. A ClickHouse node in that state may serve queries more slowly and absorb merges more slowly, which matters most on a single-replica shard. Ask your vendor for the expected rebuild time on your drive size and controller, and decide in advance whether a degraded node should be taken out of query rotation.
Monitoring disk health and RAID alerts
Controllers often report a degraded array only in their own management tool, and a disk can fail slowly through rising error counts before it drops out. Route SMART data and controller alerts into the same monitoring system that watches ClickHouse, and test that a simulated disk failure produces a page. Confirm with your vendor which tool exposes predictive failure flags for your controller model.
Controller cache and power-loss protection
A write-back controller cache speeds up small writes, but unprotected cache contents can be lost during a power failure. Confirm that the controller has a battery or flash-backed cache protection that is tested on a schedule, and that write-back mode is disabled automatically when protection fails. Ask the vendor how cache policy changes are logged and who approves them.
Encryption at rest decisions
Encryption can be applied in the controller, in the operating system, or in ClickHouse itself, and each choice affects key storage, rebuild behavior and recovery. Decide with your security team where keys live, who can unlock a host after a reboot, and how keys are rotated and escrowed. Check that the chosen method is compatible with your backup target, since backups inherit the encryption model you set up.
Access to storage hosts
Anyone with console or controller access can read or alter the data on a ClickHouse host, which is outside the database access controls. Limit shell and out-of-band management access to named administrators, record sessions, and separate controller management credentials from application credentials. Review these grants with your security team on the same cycle as other privileged access.
Replication still required on top of RAID
RAID protects against a disk failure but not against deletion, corruption or loss of a whole host. ClickHouse recommends replication regardless of RAID, and replicas recover missing data from a healthy peer once a node returns. Before go-live, confirm the replica count, the Keeper topology and the backup schedule, and test a restore on a spare cluster rather than assuming the array is enough.
Questions teams ask about ClickHouse storage layout before hiring help
Does RAID replace replication for ClickHouse?
No. RAID protects against a failed disk inside one server, but it does not protect against a failed server, a bad upgrade, an operator who drops a table, or data corruption that spreads to every copy. ClickHouse documentation and the OSS usage recommendations both say to replicate regardless of RAID. In practice, the design is RAID for disk failure on each node, ReplicatedMergeTree with ClickHouse Keeper for node failure, and tested backups for human error and corruption. Each layer covers a failure the others cannot, so removing one to save hardware usually shifts the risk rather than removing it.
What drives the cost and effort of a ClickHouse storage design?
The main drivers are the number of servers, the number of disks per server, the write profile of your data, and how many environments need the same layout. Effort rises when the team must run benchmarks on representative data, compare RAID-10, RAID-6 and RAID-50 layouts, and document the choice for the platform team. Costs split into hardware or cloud volumes, which recur as you add capacity, and design, build and verification work, which is largely one-time. Ask any provider to price the benchmark plan, the hardware validation, and the runbook separately so you can see which parts you could keep in-house.
Can our team choose and benchmark the storage layout alone?
Yes, if you have engineers who can run controlled disk tests and who have access to production-like hardware. The decision itself is not complicated: ClickHouse recommends RAID-10 with the far layout when budget allows, RAID-6 or RAID-50 instead of RAID-5 with more than four disks, and a 1 MB stripe size because average writes are about 1 MB. The harder part is proving the choice under your own load, which needs representative data, realistic query and insert patterns, and a restore test. Teams that lack that test environment often benefit from an outside review of their benchmark method before they commit to hardware.
How long does a ClickHouse storage design take, and what does it depend on?
Duration depends on how many hardware models you must compare, how much representative data you can supply, how long hardware procurement and rack access take, and whether the design must cover several sites. A single cluster on hardware already in the building is a much smaller job than a multi-site rollout with new servers. Confirm the schedule in the written scope, including which steps wait on your hardware delivery, your change windows and your approval cycles. Timelines that do not name these dependencies are usually optimistic.
What must our team provide before storage work starts?
Start with a written hardware list: server models, disk types and counts, controller models, and the filesystem you plan to use. Next, provide representative data and query logs, or a sample that reflects your ingest rate, row sizes and query shapes, because a benchmark on synthetic data can point to the wrong layout. Also arrange access windows for disk tests, since benchmarks can saturate I/O and need quiet time on the hardware. Finally, name the people who can approve a change to controller settings, and confirm which backups and restore environment are available for the restore test.
Is NFS acceptable for ClickHouse data storage?
Treat NFS with caution and confirm the details with your storage vendor and the ClickHouse documentation before you rely on it for live data. Network filesystems add latency and failure modes that local disks do not have, and the performance and locking behavior depends on your server and mount options. Local SSD or HDD with RAID, plus replication across nodes, is the layout the ClickHouse guidance centers on. If your environment forces shared storage, the design should include a written test of failover and restore, and a clear view of what happens when the mount stalls.
Should we pick SSD or HDD for ClickHouse servers?
The answer depends on your query latency needs, data volume and cost per terabyte, and you should decide it with benchmark results rather than assumptions. SSDs generally give lower latency for random reads and merges, which helps interactive dashboards, while HDDs can hold large retained volumes at lower cost per terabyte. Many designs use SSD for hot data and HDD or object storage for older partitions, which links to the storage tiering work. Whatever you choose, the RAID layout, stripe size and filesystem decisions still apply to each disk group, so document the choice per tier.
How DeployOpen scopes and delivers ClickHouse storage design
DeployOpen treats the disk layout as one part of a wider ClickHouse design that includes replication, backups and retention. We scope the storage decision against your write pattern, data volume and failure tolerance, then build, test and document the result inside your own infrastructure. Engagements are fixed-scope, with the scope agreed in writing before work starts.
How we scope
- Review your ingest rate, typical write sizes, data volume and retention targets before recommending a disk type.
- Document the RAID level, stripe size and filesystem options against your server count and disk count.
- Confirm whether NFS or other network storage is in scope, and record the caveats that apply to it.
- Write the scope so it names the hardware, operating system and ClickHouse versions covered.
How we build
- Configure and benchmark the candidate disk layouts on representative hardware before the production build.
- Set up the chosen RAID and filesystem with the stripe size and mount options agreed in the design.
- Validate that ClickHouse replicas, Keeper nodes and backups are planned on top of the storage layer.
- Test failed-disk and rebuild scenarios, and record how long replicas take to resync in your environment.
How we hand over
- Deliver a written storage design covering layout, sizing assumptions, filesystem settings and monitoring points.
- Provide runbooks for disk replacement, RAID rebuilds and adding capacity to an existing server.
- Walk your operations team through the design and the restore tests that prove the backups work.
- Record the decisions and their trade-offs so future upgrades can be reviewed against the original reasoning.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Review storage performance and capacity as data volume grows, and recommend changes before they become incidents.
- Revisit the layout when ClickHouse versions change, checking the changelog for effects on storage behavior.
- Confirm each change in a pre-production environment with representative data before it reaches production.
