How many Ceph monitors you need and where to place them
Most production Ceph clusters run three or five monitors, always an odd number, with each monitor on a host in a different failure domain. Three is the usual starting point for a single-site cluster. Five makes sense when you want to keep quorum through two simultaneous monitor losses, or when the cluster spans more rooms or sites and you want a monitor in each. Even numbers add cost without adding fault tolerance, because quorum requires a strict majority of the monitors in the monitor map.
Monitors (ceph-mon) hold the authoritative cluster maps, including the monitor map, OSD map, and CRUSH map that every client and OSD uses to find data. They agree on changes through a Paxos-based quorum. If the monitors cannot form a majority, the cluster stops accepting map changes, so placing them carelessly turns a single rack failure into a full outage. Monitors are lightweight compared with OSDs, but their availability is what keeps the rest of the cluster usable.
Placement matters as much as the count. Put each monitor on a separate host, and where possible in a separate rack, row or room that matches the CRUSH failure domains you plan to use. Use the ceph orch apply mon placement with explicit host labels or a host count so cephadm does not stack monitors on the same machine. Monitor disks should be on durable local storage with enough free space for the monitor store, because a full monitor disk is a known source of warnings. Check the health warnings for monitor disk space and set alerts before they become urgent.
Clock synchronization is the other check that teams often skip. Monitors compare timestamps, and clock skew between them produces health warnings and can disrupt quorum, so run NTP or chrony on every monitor host and verify it before handing the cluster over. Confirm the exact monitor and cephadm settings for your release against the Ceph cephadm documentation before you finalize the design.
Three versus five monitors: which count fits your failure domains
Three monitors is the usual starting point for a production cluster that spans three independent failure domains, such as three racks, three rooms or three hosts on separate power feeds. The monitors keep the cluster map, and a majority must agree before the map can change. With three monitors, the cluster keeps quorum after losing one monitor. Losing two at the same time breaks quorum, and the cluster cannot change its maps until a second monitor returns, so recovery planning matters more than the monitor count itself.
- Failures survived: one monitor or one failure domain at a time. Two simultaneous losses break quorum.
- Placement: one monitor in each of three separate domains, on different hosts with separate power and network paths. Two monitors in one rack can fail together.
- Hardware cost: three small hosts or virtual machines with reliable local storage for the monitor store. Monitors need modest CPU but are sensitive to slow disks.
- Operations: the fewest hosts to patch and monitor. During maintenance on one monitor, there is no margin left for a second fault.
- Move to five when: you need quorum to survive a planned outage in one domain plus an unplanned failure in another, or your site has grown to five independent domains.
Before you commit, check that each monitor host sits in a separate CRUSH location and that monitor store disks are not shared with heavy workloads.
Five monitors make sense when you have five or more independent failure domains and want quorum to survive two monitor losses at once. The extra margin matters most when maintenance overlaps with an unplanned fault, or when a site outage is expected to last long enough that a second failure is plausible. The benefit depends entirely on placement. If five monitors share three rooms, the room losses still remove two monitors at once, so the extra monitors add cost without adding much protection.
- Failures survived: up to two monitors at once, but only when the two lost monitors are in different failure domains.
- Placement: one monitor per domain across five or more domains. Five monitors packed into three rooms do not gain much over three monitors spread across those rooms.
- Hardware cost: two more monitor hosts, their storage, their network connections and their share of patching and monitoring.
- Operations: more hosts to watch for clock drift, disk space and image updates. Runbooks must cover maintenance on one monitor while another is down.
- Move from three when: planned domain maintenance must coexist with an unplanned failure, or your footprint has reached five or more independent domains that you can name and verify.
Ask your team to record which domain each monitor occupies, then confirm that no two share a power feed or a top-of-rack switch.
A single monitor is suitable only for labs, proofs of concept, training and rehearsing upgrades. A cephadm bootstrap creates a single-node cluster with one monitor and one manager, which is useful for learning the tooling before you add hosts. The cluster has no monitor failover, and a failed monitor stops cluster map changes until it returns. Treat any data on this layout as disposable unless you have a tested way to restore it.
- Failures survived: none. A monitor failure stops map changes, and the host is also a single failure domain.
- Placement: one host, so every component shares the same fate.
- Hardware cost: minimal, which makes it practical for learning and testing.
- Operations: easy to rebuild, but a rebuild is not a recovery plan for production data.
- Move to three when: the first real data or client workload arrives. Do not wait for the layout to prove itself under load.
If you use a lab for rehearsals, confirm that the same procedures run on a three-monitor layout before you rely on them, as the cephadm documentation covers how monitors are placed as hosts are added.
Monitor configuration options and when each fits
This table compares the main ways to count and place Ceph monitors, from three or five monitors to colocated or dedicated monitor hosts and rack or room spread. Read each row against your failure domains and the hosts you already plan to buy. The main takeaway is that monitor count sets how many failures the quorum survives, while placement decides whether one event can take out more than one monitor.
| Option | When it fits | Trade-off |
|---|---|---|
| Three monitors on three hosts | Small clusters where each monitor can sit on a different host and no rack or room holds two of them | Survives one monitor loss. Two simultaneous monitor losses stop quorum, so the cluster stops accepting map changes until quorum returns. |
| Five monitors across hosts | Larger clusters or sites that want to ride out two monitor failures at once | Needs five hosts or placements with reliable disk and network. More daemons add more quorum traffic and more places to confirm health. |
| Even monitor count | Rarely a good fit; an even count adds a member without adding a failure tolerance step | Quorum still needs a majority, so the extra member gives no better tolerance than the odd count below it. Confirm against your own layout before choosing. |
| Monitors spread across racks or rooms | Sites with separate racks, rows or rooms that can lose power or network independently | Keeps one event from removing a majority, but needs low-latency links between locations. Confirm latency and quorum behavior with your network team. |
| Monitors colocated with OSD hosts | Smaller footprints where a dedicated monitor host is not justified | Monitor disk and CPU compete with OSD recovery load. A failed OSD host can take a monitor down with its OSDs. |
| Dedicated monitor hosts | Clusters that want monitor availability independent of OSD host failures or heavy recovery | Adds hosts and their operating and patching work. Monitor hosts still need reliable local disk and time synchronization. |
| Placement with cephadm labels or placement specs | Clusters where you want monitors pinned to specific hosts or labeled racks at deployment | Labels and placement rules must be kept accurate as hosts change. Confirm the placement syntax for your cephadm release in its documentation. |
Placing monitors during cephadm deployment
This sequence turns a host inventory into a running set of monitors and managers that sit across the failure domains you planned, with each placement confirmed before the cluster is handed over. Each step closes with a check, so a gap in host layout or quorum is found while changes are still cheap.
Confirm the host inventory and failure-domain layout
Before touching cephadm, the architect or platform engineer lists every host that will run monitors, with its rack, room or site and its network reachability from the other candidate hosts. Each host should map to one failure domain in the CRUSH hierarchy you designed, so no two monitor hosts share a power feed or top-of-rack switch unless you have accepted that risk in writing. The step is done when the inventory is a single document that every monitor host appears in, with its failure domain filled in and no blank fields.
Choose the monitor count for the cluster
Decide between three and five monitors based on how many failure domains you can truly separate and how many monitors you can lose while keeping quorum. Monitor counts are kept odd so a majority is always clear after a split. Record the decision and the reason next to the inventory. The step is done when the chosen count is written down alongside the number of failure domains it covers, and that number is at least as large as the count of monitors.
Assign admin labels to the chosen hosts
Apply the
_adminlabel to the hosts that should receive the cluster configuration and admin keyring, using the orchestrator host label commands described in the cephadm documentation. Keep this set small and include hosts from more than one failure domain so that configuration remains reachable if one site is down. Run the host listing command and confirm the labels appear on the intended hosts only. The step is done when that listing matches your plan exactly.Set monitor and manager placement
Apply a placement specification for monitors that names the chosen hosts explicitly, and a separate placement for managers, so the orchestrator does not pick hosts on its own. Pinning by host name is more predictable than a count alone when hosts differ in network or power layout. Confirm the specification is accepted and that the orchestrator shows the expected number of monitor and manager daemons queued for deployment. The step is done when the planned placement is visible in the service listing with no unexpected hosts.
Deploy and watch daemons come up
Let cephadm add the monitor and manager daemons on the target hosts, either as part of host addition or by applying the placement you just set. Watch the daemon listing until each monitor reports running and each host shows its containers healthy. If a host cannot start its daemon, check container image access and SSH connectivity before retrying. The step is done when every planned monitor daemon is running and none is in an error or pending state.
Verify quorum and monitor placement
Check that the monitor map lists the expected number of monitors, that a quorum exists, and that the monitors are located on the hosts and failure domains you planned. Confirm the cluster health report does not show a monitor quorum warning. Then verify the failure-domain mapping matches the inventory from the first step. The step is done when quorum is present with all planned monitors in it and the host-to-failure-domain mapping matches your document.
Check disk space and clock sync before handover
Confirm each monitor host has free space on the filesystem holding the monitor store with enough headroom that the monitor will not raise a low-space warning, and that time synchronization is active and reporting offset on every monitor host. Cephadm will surface clock skew as a health warning, so read the health output rather than assuming it is clean. Ask the team responsible for each host to confirm the values in writing. The step is done when the health output shows no monitor disk or clock warnings and each host's time source is confirmed.
Record placement and hand over
Save the final monitor and manager placement, the admin label assignments, and the health output from the checks above into the handover pack. Include the host-to-failure-domain map so the operations team can tell which monitor loss a site outage would cause. Ask the operations team to repeat the quorum check on their side. The step is done when the handover pack is signed off and the operations team has run the quorum and health commands themselves.
Monitor disk space and clock synchronization risks
Monitors fail quietly in two ways: their stores grow until a disk fills, and their clocks drift until quorum becomes unreliable. Both risks are cheap to plan for at build time and expensive to discover during an outage, so check them before the cluster takes production data.
Monitor store disk growth
Each monitor keeps the cluster maps in a local store on its host, and that store grows while the cluster is unhealthy or while placement groups are moving. Operators see health warnings about monitor store size before the disk is actually full, so the warning is the time to act. Confirm the store location on each monitor host, the free space on that filesystem, and whether your alerting forwards monitor health warnings to the on-call team.
Clock synchronization across monitors
Monitors compare timestamps to agree on quorum and cluster state, so clock skew between monitor hosts produces clock skew warnings and can disrupt elections. Confirm that every monitor host runs a time daemon such as chrony or systemd-timesyncd, that all hosts point to the same internal time sources, and that the sources are reachable in an air-gapped site. Test skew after any reboot or network change.
Dedicated disks for monitor stores
Monitor stores share the host with the operating system, logs and sometimes OSD metadata, which means a busy shared disk can slow monitor writes. A dedicated or lightly loaded disk for /var/lib/ceph keeps monitor latency predictable and separates monitor failures from system disk failures. Ask the hardware team which disk each monitor host uses for its store, whether that disk is enterprise-class, and whether it is covered by the same replacement process as the OS disk.
Monitor reachability on msgr2
Monitors talk to each other and to clients on the msgr2 port, which is 3300 by default for the v2 protocol. A firewall rule, VLAN misconfiguration or asymmetric routing between monitor hosts can split quorum even when every host is up. Confirm that each monitor host can reach every other monitor on the msgr2 port, both from the public network and from any admin host that runs management commands, and that the rules are written into your change records.
Monitoring and rehearsing failures
Disk and clock problems are easiest to catch when they are alerted and rehearsed before go-live. Plan checks for monitor store growth, clock skew and monitor down events, and run a controlled test where one monitor host is stopped and restored. Record the expected warnings, who receives them and the steps for each, so the team does not learn the procedure during a real incident. Ask your partner or internal team to include these checks in the handover pack.
Questions teams ask about Ceph monitor placement
Is three Ceph monitors enough for production?
Yes, for many production clusters three monitors is the usual starting point, provided each monitor sits on a separate host and a separate failure domain. Monitors form a quorum, so a cluster with three monitors keeps running when one fails, but it cannot keep quorum after losing two. Whether three is enough depends on how many failure domains you can reach, how much risk you accept during maintenance, and whether you plan to add sites later. Confirm the monitor placement rules for your release in the Ceph cephadm documentation before you commit to a layout.
How does monitor count change cost and effort?
Monitors are light services, so the main cost is not the monitor software but the hosts and failure domains they need. Each extra monitor usually means another host or rack that is kept separate from the others, with its own power, network path and operating system to patch. Going from three to five monitors adds quorum tolerance but also adds hosts to build, harden, monitor and maintain. Cost drivers to examine in any quote include host count, whether monitors share hardware with OSDs or managers, the number of rooms or sites involved, and the effort to keep every node's clock and disk space in order.
What does the work need from our internal team?
Your team needs someone who can own the hosts that carry monitors, approve placement and network changes, and respond to health warnings. In practice this means access to the Linux hosts for time synchronization and disk checks, the ability to add or move hosts in your rack and network plans, and a change process for maintenance windows. Storage engineers should understand quorum and how cephadm places daemons. Security teams should confirm which hosts may run management services. Ask the implementer to document each decision so that the next operator can read the placement without reconstructing it.
Can our team set up monitor placement without a partner?
Many experienced storage teams can place monitors alone, especially if they already run Linux clusters and have tested cephadm in a lab. The work is manageable when the design is small and the failure domains are clear. Teams with little Ceph experience usually find the harder part is the decisions around failure domains, clock sources and disk thresholds, and how those choices interact with pool replication. A partner can review your design, run the bootstrap checks and hand over runbooks. Whichever path you take, test a monitor failure and a clean reboot before production use.
How long does a placement review take?
The length of a review depends on the number of hosts, whether you already have a rack and network plan, how many failure domains you must map, and how many people must approve the design. We do not publish a fixed duration for this review. Ask for a written scope that lists the inputs needed from your side, such as host inventory, rack and room layout, network diagrams and change approval steps, and the outputs you will receive. The scope and timeline are agreed in writing before work starts.
How do we move monitors to different hosts later?
Monitors can be added and removed on a running cluster, but the order of operations matters because each change affects quorum. The usual approach is to add a new monitor on the target host first, confirm it joins the quorum and syncs, then remove the old monitor. Never remove a monitor when it would leave the cluster without a quorum of the remaining monitors. Before any move, check cluster health, confirm clock synchronization on every host and make sure the monitor data store has enough free disk space. Confirm the exact commands for your release in the cephadm documentation and rehearse the move in a non-production cluster first.
What checks should we run before handing over monitors?
Before handover, confirm that the cluster reports a healthy quorum, that every monitor is on a separate failure domain, and that the expected number of monitors is running. Check that all hosts synchronize their clocks to the same trusted source, because clock skew produces health warnings and can disrupt quorum. Verify free space on the filesystem holding monitor data and set alerts before it runs low. Finally, test a single monitor restart and a host reboot in a maintenance window, and record the results in your runbook.
How DeployOpen scopes Ceph monitor and cluster design work
DeployOpen plans and implements self-hosted Ceph inside your own infrastructure, whether that is your cloud accounts, data centers or an air-gapped network. For monitor placement, that means the work covers more than picking a number. A scoped engagement typically reviews how many monitors the cluster needs, which hosts or racks will carry them, how each location maps to a CRUSH failure domain, whether the monitor hosts have enough free disk for the monitor store, and how clock synchronization is handled across every monitor host. Those decisions interact with your failure domain design and network layout, so they are scoped together rather than as isolated tasks.
Before any build starts, the scope, deliverables and acceptance checks are written down in a fixed-scope agreement. For a monitor and cluster design engagement, the document should list the hosts and sites in scope, the monitor placement diagram you will receive, the cephadm settings that will be applied, the health checks run before handover, and what is excluded, such as application-side configuration or work on hardware you have not yet purchased. Ask for the written scope to name who performs each check and what evidence you receive at the end. Timelines and effort should be stated in that agreement only after the environment has been reviewed, so ask for them to be tied to the agreed scope rather than to a generic estimate.
Monitor placement changes over the life of a cluster as you add sites, racks and hosts, and the people who run Ceph day to day need to keep the design current. DeployOpen can support the cluster after handover through a retainer, covering health review, monitor and host changes, upgrade planning and responses to warnings such as monitor disk space alerts. Confirm with us which parts of that support are included in the retainer and which are billed separately before you sign, and confirm how your own team will be trained to read the cluster state between reviews.
