One storage cluster, three interfaces
Ceph stores data in RADOS, a distributed object store that replicates or erasure-codes data across many servers and recovers automatically when a disk or host fails. On top of RADOS it offers block devices (RBD), a POSIX file system (CephFS) and an object gateway (RGW) with S3- and Swift-compatible APIs. Most of Ceph is licensed under LGPL-2.1 or LGPL-3, with some files under other open source licenses.
Ceph suits organizations that need storage capacity and resilience on commodity servers, often as the backend for Proxmox VE, OpenStack or Kubernetes. It rewards careful design: hardware, networks, failure domains and capacity thresholds decide how a cluster behaves during a failure. A small cluster built without that planning can be harder to run than the array it replaced.
Daemons and where they run
Each daemon type has a different job and failure behavior. Place them so one host or rack failure does not remove a majority of any group.
| Daemon | Role | Placement |
|---|---|---|
| MON | Maintains the cluster maps and quorum. | Run an odd number, commonly three or five, on separate hosts or racks. |
| MGR | Runs the dashboard, orchestrator and monitoring modules. | One active instance and at least one standby. |
| OSD | Stores data on a device and handles replication, recovery and scrubbing. | One per disk, using BlueStore. Spread evenly across hosts so recovery load is shared. |
| MDS | Manages CephFS metadata. | Needed only for CephFS. Run standbys and add active daemons only when metadata load requires it. |
| RGW | Serves the S3 and Swift APIs. | Run several instances behind a load balancer. They keep their data in RADOS, so instances can be added or replaced. |
Match the interface to the consumer
RBD provides thin-provisioned block devices with snapshots and clones. It backs virtual machine disks in Proxmox VE and OpenStack, and persistent volumes in Kubernetes through the Ceph CSI driver. RBD mirroring can replicate images to a second cluster for disaster recovery.
CephFS provides a shared POSIX file system for workloads with many readers and writers, such as home directories, build caches or shared application data. It needs MDS daemons and separate metadata and data pools. Plan quotas, snapshots and client access by path.
RGW implements a large subset of the Amazon S3 API and the OpenStack Swift API over one shared namespace. It supports users and access keys, OpenStack Keystone integration, an STS API, bucket notifications and multisite replication between zones. Test the specific S3 features your applications use.
Design failure domains and data protection
These decisions are hard or impossible to change once pools hold data.
CRUSH failure domains
CRUSH rules decide where replicas or chunks land. Set the failure domain to host or rack to match how your hardware fails, including shared power and switches.
Replication
Three-copy replicated pools are the common choice for block storage and metadata. They use three times the raw capacity and recover quickly. Keep min_size above one.
Erasure coding
Pools split data into k data and m coding chunks; 4+2 uses 1.5 times raw capacity. The profile cannot change after creation, m=1 is discouraged for production, and the cluster needs at least k+m failure domains, ideally one more.
Network separation
Use a public network for clients and a separate cluster network for replication where possible, sized for recovery after a host failure.
Capacity thresholds
By default Ceph warns at 85 percent (nearfull), stops backfill at 90 percent and stops client I/O at 95 percent (full). Leave room to lose the largest host and still recover.
Deploy with cephadm or Rook, then secure access
cephadm, introduced in Ceph Octopus (15.2), runs Ceph daemons in containers, manages them through the orchestrator and handles upgrades. It suits dedicated storage hosts. Rook runs Ceph as a Kubernetes operator, and Proxmox VE has its own integrated Ceph management. Choose one management path per cluster.
Use host bus adapters in place of RAID controllers, one OSD per device, and faster flash for BlueStore DB and WAL when data devices are HDDs. The default OSD memory target is 4 GiB, and recovery uses more.
Issue a cephx key per client with the narrowest capabilities, for example limited to one pool or one CephFS path. The dashboard supports SAML 2.0 single sign-on and, under cephadm, OAuth2 through oauth2-proxy, with roles assigned inside Ceph. Enable TLS on the dashboard and RGW, consider msgr2 secure mode for traffic between daemons and clients, and use OSD encryption where data at rest must be encrypted.
Moving off community MinIO to RGW
The community MinIO GitHub repository was archived on April 25, 2026 and is now read-only, and its README states that the legacy binaries will not receive updates. Teams running the community edition need a maintained S3-compatible store. Ceph RGW is one open source option, and it fits well where Ceph already provides block or file storage.
Inventory what applications use: bucket policies, versioning, object lock, lifecycle rules, presigned URLs, server-side encryption, notifications and the identity model behind access keys. RGW implements a large subset of the S3 API, so test each client library and workflow against a staging RGW before moving data.
Copy buckets with an S3-aware tool such as rclone, verify object counts and checksums, then switch endpoints and credentials one application at a time. Keep the MinIO deployment read-only until owners confirm that their data and access work on RGW.
Snapshots, backups and upgrades
RBD and CephFS snapshots give fast recovery points, and RBD mirroring or RGW multisite can keep a copy in a second cluster. They protect against local mistakes and site loss, but they share software and often credentials with the source cluster.
Replication inside Ceph copies deletions and corruption as faithfully as good data. Keep backups of critical data on a separate system with separate credentials, and test restores into a clean pool or bucket.
As of October 2026, the Ceph releases page lists two actively maintained releases, Tentacle (20.2) and Squid (19.2), with an estimated end of life for Squid of October 31, 2026, so Squid clusters should plan their move. cephadm upgrades daemons in a defined order while the cluster keeps serving data. Read the release notes for supported starting versions and confirm the cluster reports HEALTH_OK first.
Ceph operating ownership
Ceph keeps serving data through many failures, which makes warnings easy to ignore. Assign people to watch them and act.
Storage operations
- Watches health checks, placement group states, scrub results and slow operations.
- Replaces failed disks and hosts and tracks recovery.
- Plans and runs release upgrades.
Capacity
- Tracks raw and usable capacity against nearfull and full ratios.
- Models the effect of losing the largest host or rack.
- Orders hardware before thresholds are reached.
Consumers
- Own the data in their pools, images, file systems and buckets.
- Request pools, quotas and keys through a defined process.
- Back up data that cannot be recreated.
Security
- Reviews cephx capabilities, RGW users and dashboard roles.
- Holds encryption keys and backup credentials under separate control.
- Tracks Ceph security advisories.
Ceph deployment questions
How many nodes do we need?
Three hosts is the practical minimum for replicated pools with a host failure domain, and the Proxmox VE documentation recommends at least three nodes and twelve OSDs for hyper-converged Ceph. Erasure coding needs at least k+m hosts, plus one more for recovery headroom.
Can Ceph replace our SAN?
For many virtualization and Kubernetes workloads, RBD can take that role. Test latency-sensitive databases on your own hardware and network before committing, since every Ceph write crosses the network to several OSDs.
Is RGW a drop-in replacement for MinIO?
Both speak S3, but feature coverage, identity and administration differ. Test every application against RGW and plan the credential change for each one.
What happens when the cluster fills up?
When any OSD reaches the full ratio, 95 percent by default, Ceph stops client I/O to protect data. Freeing space at that point is slow and risky, so alert on nearfull and add capacity early.
Scope and handover
Effort depends on capacity and growth, the interfaces in use, hardware and network design, failure domains, encryption needs, integration with Proxmox VE, OpenStack or Kubernetes, and any data migration such as a move from MinIO. Hardware selection and network design decide much of the outcome before any software is installed.
DeployOpen plans the cluster, deploys it with cephadm or Rook, hardens access, connects dashboard SSO, sets up snapshots, replication and off-cluster backups, runs upgrades and documents the result. Handover includes the hardware and network map, CRUSH rules and pool layout, capacity model, key and capability inventory, backup and restore evidence, the upgrade runbook and a support route.
