Why bootstrapping a production Ceph cluster with cephadm needs planning first
cephadm manages a Ceph cluster as containers. It bootstraps a single-node cluster with one Monitor and one Manager, then adds hosts and services to grow that starting point into a production cluster. cephadm was introduced in v15.2.0 (Octopus) and does not support older releases, so the first decision is confirming that your target release is one it supports. The actively maintained releases are Squid (19.2.x) and Tentacle (20.2.x). Check the Ceph releases page for current point releases and end-of-life dates before you pick one, because a cluster built on a release near the end of its support window will need an upgrade sooner than you may plan for. Confirm the exact version with your team's change process, not from memory.
Bootstrap is the easy step to repeat and the hard step to undo. The bootstrap command sets the first monitor's IP address, the cluster's identity and the initial admin credentials, and the hosts you add afterward inherit the network and naming conventions you chose at that point. Changing the public network, moving the first monitor, or renaming hosts later means editing live cluster configuration, which is work you should avoid by deciding these items in advance. Before running anything, write down the public network, the optional cluster network, the hostnames, the SSH user cephadm will use for host onboarding, and which hosts will carry the admin label that distributes the configuration and keyring. Also confirm that each host meets the requirements in the Ceph cephadm documentation for container runtime, time synchronization and free disk space before you start.
Which bootstrap options to set and when each fits
This table compares the choices you make when running cephadm bootstrap and what each one costs to change later. Read each row as a decision to settle before the first monitor starts, since several settings are hard to unwind once hosts and services depend on them. Confirm flag names and defaults against the cephadm documentation for your Ceph release before you script the bootstrap.
| Bootstrap choice | When it fits | What is costly to change later |
|---|---|---|
| Monitor IP address | Set explicitly when the first host has several interfaces, so monitors bind to the network your clients and peers will reach. | Changing the monitor address means re-addressing monitors and redeploying their configuration, which is disruptive on a live cluster. |
| Cluster network at bootstrap | Choose it when replication and recovery traffic should run on a separate bonded network from client traffic. | Adding or moving the cluster network later requires configuration changes and OSD restarts, so plan the subnet before any OSD is created. |
| SSH key and user for host additions | Keep the default key pair generated at bootstrap for a single management host, or supply your own key and a dedicated user when your security team manages SSH access. | Rotating the key or changing the user after hosts are added means updating every host, so agree the SSH identity with your security team first. |
| Dashboard credentials | Set an initial administrator through your secrets process and confirm how the Dashboard will later integrate with your single sign-on, if you plan that. | Reworking authentication after users and automation depend on local accounts can break integrations, so confirm the login model before handover. |
| Local or mirrored container image | Point bootstrap at an internal registry image on air-gapped or restricted networks, and check that monitoring and Ceph images are mirrored too. | Switching image sources later means redeploying daemons, and an out-of-date mirror will block upgrades, so keep the mirror on a schedule. |
| Applying a specification file at bootstrap | Apply a specification when the host layout and service placement are known and reviewed, so monitors and managers land where you designed them. | Placement made at bootstrap is easy to edit, but unexpected daemons on the wrong hosts add cleanup work, so dry-run specifications first. |
What each host must provide before bootstrap
Every host in a cephadm cluster runs Ceph daemons as containers and must be reachable, time-aligned and wired the same way as its peers before bootstrap starts. The six items below are the checks that most often decide whether bootstrap and host addition go smoothly, so confirm each one on every host, not only on the first.
Supported container runtime
cephadm runs every Ceph daemon as a container, so the host needs a container runtime that the cephadm documentation supports for the release you chose. Confirm the runtime, its version and that the admin host can pull images from your chosen registry. Check the exact requirements in the cephadm documentation for your release before you standardize the operating system image.
Consistent hostnames and name resolution
cephadm identifies hosts by name when it adds them and places daemons on them, so each hostname must match on the host and resolve the same way from every other host and the admin node. Use forward and reverse DNS or identical /etc/hosts entries, and avoid short names that collide across sites. A mismatch here usually shows up later as hosts that appear offline or daemons that will not place.
Time synchronization across hosts
Monitors depend on clocks that agree closely with one another, and clock skew produces health warnings that look like storage faults. Run one time service on every host, point all hosts at the same internal sources, and confirm sync status before bootstrap. Check the monitor clock guidance in the Ceph documentation for the tolerances that apply to your release.
SSH reachability from the admin host
cephadm manages hosts over SSH from the node where you run bootstrap, so that node needs key-based root or sudo-capable access to every host it will add. Confirm that the SSH keys work, that host firewalls allow the connection and that no bastion step is missing. Record which account cephadm will use, because access policy changes later affect how the cluster is administered.
Dedicated drives for OSDs
The Ceph hardware recommendations advise a dedicated drive per OSD, so the operating system and any other workload must not share the disks Ceph will consume. Keep system disks separate, and plan enterprise-class SSDs for metadata and BlueStore DB or WAL where your design calls for them. Confirm drive models and firmware against your vendor's compatibility list before ordering.
Network links bonded across two switches
The hardware recommendations advise bonding network links across two switches, so one switch failure does not isolate a host from the cluster. Agree the bond mode and the public and cluster network subnets with your network team before bootstrap, and test that every host reaches the others on both networks. Memory should also be sized for peak use rather than calm periods, which the same guidance recommends.
Build sequence from the first monitor to a healthy cluster
This sequence takes bare hosts to a cluster with monitor quorum, standby managers and admin keyring distribution on chosen hosts. Each stage closes with a check you can run and record, so faults surface while the cluster still has only a few daemons.
Bootstrap the first host
Run
cephadm bootstrapon the first host with the monitor IP address and the dashboard and network options settled in your design decisions. cephadm starts one Monitor and one Manager as containers, then writes the initial configuration and admin keyring on that host. The step is done whenceph -sreturns status from the new cluster andceph orch pslists the Monitor and Manager as running on that host. Save the bootstrap output, because it holds the dashboard credentials you will need.Distribute the cephadm SSH public key
Retrieve the cluster public key with
ceph cephadm get-pub-keyand install it for the SSH account that cephadm uses on each new host. Your platform or security team typically performs this step, because it grants the orchestrator control over the host. Confirm that the key works by running a connection check against each host before you add it to the cluster. A host is ready when the check passes without prompting for a password.Add each host over SSH
Add hosts one at a time with
ceph orch host addusing the hostname and address. cephadm connects over SSH, installs its agent and starts the daemons it needs on that host. Confirm that the host reports as online inceph orch host ls, and check that its container runtime, time synchronization and network interfaces match your design. Finish this step for every host before you move on, so that later placement decisions use the full inventory.Assign the admin label to selected hosts
Apply the
_adminlabel withceph orch host label addto the hosts that should hold the admin keyring and configuration. cephadm copies both files to every labeled host. Choose a small number of hosts that operators actually use, and treat each one as privileged. The step is done whenceph orch host lsshows the label on the intended hosts and the keyring file exists under/etc/cephon each of them.Place monitors and managers with explicit placement
Use
ceph orch apply monandceph orch apply mgrwith explicit host lists rather than letting the orchestrator choose. Spread monitors across the failure domains you designed, and use an odd number of them so quorum survives the loss of one. Place a standby Manager on a second host so the dashboard and orchestrator remain available during failover. Check thatceph orch ps --daemon-type monand--daemon-type mgrshow each daemon on its intended host, and thatceph mon statreports quorum with the expected members.Confirm health and daemon placement before handover
Run
ceph -sandceph health detail, then resolve or document every warning before the cluster is handed to operations. Check thatceph orch lsshows each service at its expected count and thatceph orch psplaces every daemon on the host you planned. Confirm that clocks are synchronized across hosts and that the dashboard answers on its management address. Save this output as the baseline for later upgrades and troubleshooting. The cluster is ready for handover when the health status is understood and every daemon matches the design.
Where monitors and managers should run in each environment
A lab cluster exists to learn cephadm, test workflows and validate configuration, not to survive a rack failure. Placement is simple because there is usually one host or a few hosts with no independent failure domains. A single-node cluster bootstraps with one Monitor and one Manager, which is enough to explore the system but gives no protection against the loss of that host. Plan the lab so that moving to production later does not require rebuilding it.
- Single host limits: Every monitor and manager shares one failure domain, so any host outage stops the control plane.
- Odd monitor count: Monitors form a quorum by majority vote, so an odd count means one monitor can fail without losing the majority; an even count adds a monitor without adding tolerance.
- Manager failover: A standby manager only helps if it runs on a different host, so a lab with one host has no failover to test.
- Deferred decisions: The monitor count and placement rules belong to the monitor placement sibling page, and failure domain layout belongs to the failure domain page.
- Keep the layout portable: Record host names, labels and the bootstrap command so the same design can be re-applied on production hosts.
Before you rely on a lab for evidence, confirm which parts of the behavior it cannot reproduce, such as failover timing across real switches and network loss.
A single rack gives you several hosts behind shared power and a shared top-of-rack switch. Monitors and managers should still be spread across hosts so that one host failure does not remove the control plane. Because the rack itself is one failure domain, a rack-level event such as a power or switch failure can take down the whole cluster, and placement inside the rack cannot prevent that. Treat this as a deliberate limit in the design record.
- Spread monitors across hosts: Place each monitor on a different host so that losing one host costs at most one monitor.
- Odd count for quorum: Choose an odd number of monitors so that a majority remains after a single monitor or host loss.
- Standby managers on other hosts: Keep at least one standby manager on a host other than the active one so the dashboard and orchestrator fail over.
- Label hosts for placement: Use admin labels and placement specifications to keep monitors on the hosts you chose, so future host additions do not move them silently.
- Deferred decisions: Monitor count, disk space for the monitor store and clock synchronization are covered on the monitor sibling page, and power and switch redundancy inside the rack is covered on the failure domain page.
Check that the monitor and manager placement survives a planned host reboot, and confirm with your facility team how power and top-of-rack switches are fed.
A multi-room or multi-site design lets monitors and managers follow real failure domains such as rooms, rows or sites. The goal is that no single room failure removes quorum or leaves the cluster without an active manager. Placing monitors in one room, even on separate hosts, concentrates risk. The placement rules must match the CRUSH failure domains chosen for the data, and the network paths between rooms determine how quorum behaves when a link degrades.
- Follow the room boundary: Distribute monitors so that the count surviving the loss of any single room still forms a majority, which usually means no room holds a majority of monitors.
- Spread managers across rooms: Keep standby managers in a different room from the active manager so that failover works when a room is lost.
- Latency between sites: Confirm the round-trip time between rooms with your network team, because monitors are sensitive to latency and clock drift.
- Quorum after a split: Test which side keeps quorum when the links between rooms fail, and document the outcome.
- Deferred decisions: Choosing the room count, failure domains and replica rules is covered on the failure domain page, and multi-site replication is covered on the Ceph features pages.
Before committing to a layout, confirm the design with the vendor documentation for your Ceph release and with an engineer who has tested the link behavior in your own facilities.
Security decisions to make during bootstrap
Bootstrap sets the access and trust model for the whole cluster before any data arrives. The six decisions below cover who can reach the orchestrator, how people sign in to the dashboard, how traffic is protected on the wire, and how bootstrap secrets are retired. Confirm each control against the Ceph release you install, because option names and defaults change between releases.
Who holds the cephadm SSH key
Cephadm reaches every host over SSH using a key pair that the orchestrator manages. Decide which people and automation can read the private key, and which hosts receive the public key. Treat the admin host as the place that can change every node, and record who can reach it. Confirm with the Ceph release notes how the key is generated, rotated and stored in your version, and test that a removed admin can no longer reach hosts.
Admin label placement
The _admin label tells cephadm which hosts receive the admin keyring and configuration file. Each labeled host can run ceph commands with full cluster rights, so apply the label only to hosts that need it, usually two or three management nodes. Review the label list after each host addition. Ask the vendor or documentation which hosts receive the admin keyring by default in your release.
Dashboard sign-in method
The Dashboard can use single sign-on through SAML 2.0. It can also use OAuth2 when cephadm runs the Management Gateway and OAuth2 Proxy services. Choose the method before users log in, because local accounts created during bootstrap are a weaker long-term control. Confirm which services your release deploys, and test sign-in, role mapping and a forced logout before handover.
Msgr2 secure mode
Messenger v2 listens on port 3300 and its secure mode encrypts all traffic after authentication with AES-GCM, adding integrity checks. Decide whether secure mode applies to every daemon and client in your design, then confirm that older clients and firewall rules allow port 3300. Verify the setting on live connections rather than assuming it from the configuration file.
Admin host access control
The admin host runs the cephadm shell and holds the keys that manage the cluster. Restrict it to named administrators with individual accounts, enable session logging, and keep it off general user networks. Decide whether administrators reach it through a bastion, and who approves access changes. Ask your team to confirm which controls apply to this host in your environment before handover.
Bootstrap credential rotation
Bootstrap creates an initial dashboard administrator password and other credentials that should not outlive the install. Record where the first password is stored, move it into your secrets vault, and set a rotation date that your change process owns. Remove any temporary accounts, and confirm the rotation steps for your release in the hardening guidance, which covers these controls in more depth.
Checks to pass before handing the cluster over
The handover gate should confirm that the cluster is healthy, complete and recoverable before anyone else depends on it. Start with ceph health detail and treat any warning that is not explained as a blocker. Then confirm that every host and daemon appears where your placement intended, using ceph orch host ls and ceph orch ps. Check that the monitors are in quorum and that a standby manager takes over when the active one is stopped, since a manager that does not fail over will leave the dashboard and orchestrator unavailable during the first real incident.
Next, verify the access and network paths. Open the dashboard through the access path you chose at bootstrap, whether that is a direct host address or a load balancer, and confirm that it responds from the networks your administrators use. Confirm time synchronization on every host, because monitor clock skew is a common source of warnings. Test host-to-host reachability on the public and, if you configured one, the cluster network, including the msgr2 port 3300 path between monitors and the other daemon ports in use.
Finally, run one controlled disruption in a maintenance window: reboot a host or fail over the active manager, then watch recovery until health returns to its baseline. Record the bootstrap configuration, including the image, the admin host, the labels, the monitor and manager placement and the network settings, so the next operator can rebuild or audit the cluster. Treat this list as a starting point. Measurable acceptance criteria, such as the exact recovery time or the warnings you will tolerate, belong in the written scope agreed before work starts.
Limits to confirm in your environment before you start
A cephadm bootstrap is only as good as the assumptions underneath it. The cards below cover the decisions and checks that sit outside the bootstrap command itself, from release lifecycle to registry access and platform choice. Settle each one in writing before hosts are racked or images are pulled.
Confirm the release against your upgrade plan
Ceph's active releases are Squid (19.2.x) and Tentacle (20.2.x), and each has an estimated end-of-life date on the releases page. Compare those dates with your own change calendar and the time you need to complete upgrades before you choose a release to bootstrap. A cluster built on a release that ends support before your next planned upgrade window forces an early upgrade. Ask your Ceph partner or your own team to map the dates to a written upgrade plan.
Confirm registry and network access for images
cephadm pulls Ceph and monitoring container images during bootstrap and when adding hosts, so the hosts need a registry they can reach. Check whether your egress policy, proxy rules and firewall allow that registry, and whether your security team requires an internal mirror. If the site is isolated, plan a local registry before the first bootstrap, not after it fails. Confirm who owns image approval and how new images enter the environment.
Verify hosts against the release requirements
Each release has host requirements for container runtime, Python, time synchronization, systemd, free disk space and network ports. Check every candidate host against the documentation for the release you chose, not an older one. Record the operating system version, kernel and container engine on each host, and confirm that SSH access for the cephadm user works from the admin host. A host that fails these checks at bootstrap will cost more time to correct than to prepare in advance.
Decide between cephadm and Rook on Kubernetes
The cephadm documentation points to Rook as the alternative for Kubernetes, and the CNCF lists Rook as a Graduated project. If your storage will serve workloads that already run on Kubernetes, Rook may fit your operating model better than a separate cephadm cluster. Compare who will operate the storage, how upgrades are scheduled alongside cluster upgrades, and how failure domains map to your Kubernetes nodes. Make this choice before bootstrap, because moving a running cluster between models is a migration project.
Check hardware and firmware support
Ceph's hardware recommendations call for a dedicated drive per OSD, enterprise-class SSDs for metadata and bonded network links across two switches. Confirm with your server and drive vendors that the chosen models, HBA firmware, NIC drivers and BIOS settings are supported for the Ceph release and the host operating system you plan to run. Ask vendors to state in writing which configurations they support, because a vendor that does not support a configuration will not help you troubleshoot it.
Assign ownership of the cluster and its images
Before bootstrap, name the people who will hold the cephadm admin role, approve image versions, schedule host maintenance and respond to health warnings. Decide how SSH keys for the cephadm user are issued, stored and rotated, and who can log in to the admin host. Unclear ownership is the most common reason a healthy cluster drifts into an unpatched state months later. Document these answers next to the bootstrap record so the handover includes them.
Questions teams ask before bootstrapping Ceph with cephadm
How much does a cephadm bootstrap engagement cost?
A bootstrap engagement is priced from a written scope, so there is no standard fee. The main cost drivers are the number of hosts, how many networks and failure domains the design needs, whether the site is air-gapped, and how much acceptance testing you want before handover. Ask for a quote that lists each deliverable separately: host preparation checks, bootstrap and host onboarding, service placement, monitoring integration, documentation and test evidence. Confirm which items are one-time work and which would move to an ongoing support retainer, since Ceph clusters need upgrades and patching after handover.
How long does a Ceph bootstrap take?
The bootstrap itself is usually the shortest part. Most of the elapsed time depends on hardware readiness: racking, drive firmware, switch configuration, bonded links and DNS or NTP being in place on every host. Acceptance testing, including failure tests such as pulling a node or a drive, also takes time because it has to run long enough to produce useful evidence. Ask for a schedule that starts from the day the hosts are reachable over SSH, and confirm which prerequisites your team owns.
What must our internal team provide before the work starts?
Your team needs to provide access and decisions that cannot be made by the implementer alone:
- Hosts with the agreed OS, time synchronization, and SSH access for the cephadm user
- Network addressing, VLANs, bonds, and the separation of public and cluster traffic
- Drive inventory and the intended role of each device
- A container image source, either public registries or a local mirror for air-gapped sites
- Named owners for monitoring, backups, identity, and change approval
Ask the implementer to list these inputs in the scope so gaps are found before the bootstrap window.
Can our own team bootstrap a production Ceph cluster?
Yes, many teams do, and the cephadm documentation covers the bootstrap and host-adding steps. The difficult parts are design decisions that are costly to change later: failure domains, placement group settings, monitor placement, network layout, and drive layout. Consider whether your team has people who have run Ceph through a failed drive, a full OSD, and an upgrade. If not, a partner can build the design, run the bootstrap, and document it so your team can operate the cluster afterwards.
Should we bootstrap Ceph with cephadm or Rook?
Choose cephadm when the Ceph cluster is a standalone storage service on hosts you manage directly, and choose Rook when your workloads already run on Kubernetes and the storage should be managed through Kubernetes objects. The cephadm documentation points to Rook as the alternative for Kubernetes environments. Ask which operating model your platform team will support, because the day-two tasks, upgrade paths, and failure recovery steps differ between the two.
How do we add hosts to the cluster after bootstrap?
New hosts are added over SSH with cephadm, after the host has the same container runtime, time synchronization, and network setup as the existing nodes. Label hosts that should receive the admin keyring and configuration, then place monitors and managers deliberately instead of accepting defaults. Confirm the host-add procedure and label naming with your team before the first expansion, and check the current cephadm documentation for the exact commands on your release.
What changes for an air-gapped bootstrap?
An air-gapped site cannot pull Ceph or monitoring images from public registries, so it needs a local registry that holds every image the bootstrap and later upgrades require, and cephadm must be pointed at that registry. Registry trust, whether plain or TLS, must be set up on every host. The same mirror has to be kept current for each upgrade, which is recurring work. Include the registry build, image transfer process, and an upgrade rehearsal in the quote, since they add effort beyond a connected install.
What should a production handover include?
A handover should give your team evidence and documentation, not only a healthy status line. Ask for the cluster health output at handover, the bootstrap and host inventory, monitor and manager placement with reasons, network and failure domain design, drive and service specifications, the list of tested failure scenarios with results, backup and monitoring configuration, and an operations runbook covering upgrades and drive replacement. Confirm which items are DeployOpen deliverables and which depend on your own monitoring and change processes.
How DeployOpen works on this
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including the cloud accounts, data centers or air-gapped networks where your Ceph cluster will run. For a cephadm bootstrap, that means agreeing the host list, network layout and admin access in writing before the first host is touched, then handing over a cluster you can operate.
How we scope
- Written scope lists the bare hosts, operating system baseline, network interfaces and bootstrap options to be configured.
- Scope names the admin host, SSH key handling, and which hosts receive the admin label and monitor placement.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before work begins.
How we build
- Hosts are checked against the cephadm prerequisites, then bootstrapped with the agreed monitor and manager settings.
- Remaining hosts are added over SSH and placed with explicit labels so monitor and manager location is deliberate.
- Each step is tested on the cluster itself, with health output and service placement recorded before moving on.
How we hand over
- Handover documents the bootstrap configuration, host inventory, labels, service placement and recovery of the admin keyring.
- Your team receives a checklist of health checks run at handover, with the expected output for each.
- Operating notes state which Ceph release line is deployed and which upgrade path applies to it.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Retainer scope should be confirmed in writing, including which hosts, releases and change windows it covers.
- Ask any partner to confirm how they will handle cephadm image mirrors for future upgrades in your environment.
